Command Line Mac

Ruby: Sorting an array with multiple fields

2011-11-17T15:01:00.001-08:00

I was working on a project today and needed to make some changes to the way an array on objects was sorted before being displayed on the screen. The main reason I even bothered to post this is to heap a little more praise on Ruby for working the way I expected it to work.

Here was my object (not tied to a database, just created for convenience):

class Force
attr_accessor :id
attr_accessor :last_name
attr_accessor :first_name
attr_accessor :rank
attr_accessor :shift
attr_accessor :promoted_on
attr_accessor :phone
attr_accessor :totalcount
end

I needed to sort an array of these objects by rank ascending, shift ascending, then promoted_on (date) descending.

I sorted them by concatenating the fields together and making the right comparisons. This only works because the first two fields, rank and shift, cannot be empty. Otherwise, a simple concatenation would create undesired results.

Sorting all fields ascending looks like this:

@forces = @forces.sort {|x,y| x.shift + x.rank + x.promoted_on.to_s <=> y.shift + y.rank + y.promoted_on.to_s}

To sort the promoted_on field in descending order, reverse the x,y sides for that field:

@forces = @forces.sort {|x,y| x.shift + x.rank + y.promoted_on.to_s <=> y.shift + y.rank + x.promoted_on.to_s}

No need for a special sort coding block. Ruby FTW!

Steve Jobs RIP

2011-10-06T09:56:00.001-07:00

Steve Jobs RIP

The Chromium OS experiment

2011-09-18T12:55:00.000-07:00

Google's Chromium OS

I downloaded a bootable ISO of the Google Chromium OS release candidate. Then, fired it up in VMware Player and took a look around. It appears to be based on OpenSUSE Linux 11.4. SUSE was one of the distributions I cut my teeth on in the Linux world. I always appreciated the 9 pound manual of documentation that came with it. Very helpful for beginners and a good reference to their tools.

I might have made a mistake when setting up the virtual machine because it complained of low memory right after booting. This could also explain the slow performance I experienced. The desktop itself is clean and appears to be optimized to run the Chrome web browser. There was a local word processing application, but I didn't get far enough to test it out, it was just too slow. I could not find a download link at Google, but I found one at a .eu domain. Maybe it was a pirated or tampered version. I am going to postpone further research until I am sure I have a good distribution.

Updated: Turns out Google doesn't provide an image or download of the OS itself, only the source code and compilation instructions. Also, the OS is based on Ubuntu, despite the build I got from Europe based on SUSE. ZD Net suggested that the running OS would be slow so I maybe I am not missing much at this stage of development. I don't want to pay for a Chromebook just to run Chrome. Another reason to keep this on hold for now.

Getting by in Git

2011-07-20T13:42:00.000-07:00

Git is the source code management system used for the Linux kernel and many other highly complex projects. It was written by Linux Torvalds after some controversy over the proprietary Bitkeeper program that used to manage the Linux kernel.

I've needed to upgrade my skills recently to use git in place of subversion because that is what my shop decided to use. I've moved all my Rails code into a remote git server and so far, so good.

One improvement is it has fewer "droppings" than subversion. There is no hidden .svn directory in each directory with source code. Only a single .git directory at the root of the project, plus a .gitignore file for files you don't want git to track.

Initialize project tracking
git init

Check out an existing project from remote server
git clone ssh://server/git/project

Add a file for git to track
git add

Add all files from this directory and below for git to track
git add .

Commit all files to local repository
git commit -a -m "message"

Undo changes to a file (re-check out from repository)
git checkout --

Pull files from remote repository and merge with local repository
git pull

Push files to remote repository (must commit first)
git push

Move file or directory to new location
git mv path destination

Remove file or directory from the working tree
git rm path

To create a remote repository from an existing project takes several steps
cd /tmp
git clone --bare /path/to/project (creates a /tmp/project.git directory)
scp project.git to remote server
cd /path/to/project
git remote add origin ssh://server/git/project
git config branch.master.remote origin
git config branch.master.merge refs/heads/master

LVM basics

2011-05-26T14:10:00.000-07:00

I've just spent a few hours with iSCSI SAN disks (EqualLogic) and Linux Logical Volume Manager (LVM). The abstraction is even deeper than that, because Linux at work is running under VMware, so it is really VMware talking to the SAN and presenting a SCSI disk to Linux. Since I only get into the LVM weeds a couple of times a year, I thought it would be helpful to list the steps I took to get usable disk space under Linux starting with the raw disk space.

Step One - create a new partition
Create a new partition with FDISK or PARTED. Mark the partition type hex 8E for LVM. In my case, the SCSI disk appeared as /dev/sdb and the partition using all space became /dev/sdb1. LVM is capable of using a raw device (no partition type), but I stayed in familiar partitioning territory.

Step Two - create LVM physical volume
pvcreate /dev/sdb1

Step Three - create LVM volume group in the physical volume
vgcreate new_volume_group /dev/sdb1

Step Four - create LVM logical volume in the volume group
lvcreate --name new_logical_volume --size 100G new_volume_group

Step Five - create a file system on the logical volume
mkfs -t ext4 /dev/mapper/new_volume_group-new_logical_volume
Note: Linux device mapper automatically creates a symlink to the disk in /dev/mapper using the volume group and logical volume names. If you choose more meaningful names than the example, the name won't look so awful.

Step Six - turn off automatic file system checks (optional)
tune2fs -c 0 /dev/mapper/new_volume_group-new_logical_volume

Step Seven - add mount point in /etc/fstab
Once the mount point is list in fstab, mount it manually and it is ready to use.

Chrome Revisited

2011-04-13T10:13:00.000-07:00

In my first serious test of Mac Chrome back in August, 2010, it came up short compared to other popular web browsers. But Chrome has been evolving fast. And since Firefox 4.0 was just released, it seemed like the right time to run another informal comparison.

What a difference. Chrome has become more stable, faster, and has made huge strides in available and useful plugins. The plugin gap with Firefox was a glaring issue last time. As a web developer, Firefox was indispensable with the Firebug plugin, letting you drill into the CSS and JavaScript acting on individual DOM components. Chrome now has native developer tools that rival Firebug. I am also fond of the Awesome Screenshot plugin that allows you to capture and annotate a web page from within the browser.

While I have found Firefox 4.0 a nice improvement in looks and rendering speed, I have also found it leaks memory, more so on Windows than Linux or Mac. I can rarely make it through a day without the Windows version locking up. That may be an artifact of the slew of plugins I am running and not the Firefox core.

For the last couple of weeks, I have been using Chrome as my primary browser on all platforms and have been very pleased. Competition is a great thing and I am excited to see browsers evolving again after what seemed like a long period of stagnation.

Android G2 and iTunes music sync

2011-01-23T09:10:00.000-08:00

Getting music from iTunes on a Mac onto an Android G2 smart phone was an easy task. Maybe it wasn't so easy with the first generation Android phones, but the G2 I purchased a couple of weeks ago came with everything I needed.

Connecting the phone to the Mac

My main Mac is a two year old MacBook. The G2 came with a USB sync cable and as soon as I attached it, it opened iPhoto to upload pictures just like a camera. But the G2 also displayed a screen to enable it as a USB disk drive. I enabled disk mode, then took a look at the mounted volume in Finder.

One of the icons on the Volume was DoubleTwist. When I double clicked it, it started downloading the latest Mac version of DoubleTwist. After installation, the DoubleTwist interface looked very much like iTunes. It allows you to automatically sync all music from iTunes or just selections.

Since I don't plan to listen to music often on the G2, I chose to only sync playlists. This only syncs the music required in each playlist. There are a lot of options in DoubleTwist and the integration was seamless.

Automating FTP

2011-01-02T20:01:00.000-08:00

People frequently need to automate FTP sessions to upload or download files.
Most command line FTP clients, including the FTP client on the Mac, can automatically login to an FTP server by reading the .netrc file in the user home directory. Note that the FTP auto-login file starts with a dot in front of the name (dot netrc).

Syntax of the $HOME/.netrc

The .netrc file can contain more than one auto-login configuration. Each FTP server has a set of commands, the minimum being the login name and password. You can create as many machine sections as you need. Here is a generic example:

machine ftp.server.com
login myuserID
password mypassword

Very Important: .netrc permissions!
Since user IDs and passwords are stored in the .netrc file, the FTP client enforces permission checking on it. It must be set so that no groups and no other users can read or write to it. You can set the permissions on it with this command from the Terminal (from your home directory) once the file is created:
chmod 700 .netrc

Adding FTP commands in a BASH script
You can embed FTP commands in a BASH script to upload and download files.
For example, you could create a script file named ftpupload.sh:

#!/bin/bash
# upload a file
/usr/bin/ftp -i ftp.server.com <<ENDOFCOMMANDS
cd backupdir
cd subdir
put datafile
quit
ENDOFCOMMANDS

In this example, I added the -i switch when running FTP to prevent it from prompting on multiple file uploads/downloads, even though it is only uploading one file in the example. I also use the BASH HERE document feature to send commands to FTP. When the script is run, it will auto-login using the information in the .netrc file, change to the right remote directory and upload the datafile.

Scheduling the script with Cron
The last step is to get the BASH script to run unattended, say every day at 5:00 am. The old school UNIX way is to use Cron, but the fancy new Apple way is to use a launchd XML configuration. As long as cron is supported in OS X, I'll stick to the old school way. I leave the launchd configuration as an exercise for the reader.

Add these lines with the command "crontab -e", then save:

# automated FTP upload
0 5 * * * /Users/username/ftpupload.sh

Dumping a Postgresql database remotely with SSH

2010-09-15T09:14:00.000-07:00

I ran into a rare problem recently with a large Postgresql database that was filling up the local disks of a server. The database was large, over 100 GB and about 300 million records. There was a lot of churn and it had not been vacuumed in a long time. When I manually ran a vacuum on it, there was not enough working disk space to complete the operation, creating a bind.

What I decided to do instead of using vacuum was to dump it to a remote backup location, then drop the database and restore it from the remote dump. I used SSH to run the remote commands.

Dump a remote Postgresql database to the local machine
ssh user@remote-database-server 'pg_dump database-name -t table-name' > table-name.sql

Restore a remote Postgresql database dump to the local database server
ssh user@backup-machine 'cat table-name.sql' | psql -d database-name

Note that the dump command is run from the backup machine and the restore command is run from the database server.

Also note the single quotes around certain parts of the command.

The Google Chrome experiment

2010-08-19T16:26:00.001-07:00

When Google first announced their Chrome browser, packed with a revamped JavaScript engine (V8) and support for offline web apps, I thought I would give it a spin. The first couple of releases were for Windows and Linux -- no Mac version. Those early versions were a little clunky and appeared to offer no better performance than other popular browsers. So I moved on.

When the Mac version became available, it was a much more polished browser. Another theoretical selling point was that each tab ran as a separate process so one crashed tab would not crash the whole browser. I decided to give Chrome a serious work out on my Mac at home.

Things started out well enough and performance was good. I perused the help and learned some of the short cuts. After about three good weeks, something went wrong. I don't know if it was an update, a growing cache, or what, but it started slowing down. Then, it started having problems loading pages from web sites that worked fine in other browsers. It is possible that it even caused wireless network issues, though that is just speculation at the moment. I need to do some more research to see if the problems were related to Chrome.

For now, I am sticking with Firefox as my main browser on the Mac. I'll come back and try Chrome out after the next major release.

Finding IPs connected to your web server

2010-07-01T16:36:00.001-07:00

On Mac OS X
note: this also shows outgoing connections from web browsers

Get all IPs connected to your web server:

netstat -nat | sed -n -e '/ESTABLISHED/p' | awk '{print $5}' | sed 's/\./ /g' | awk '{print $1"."$2"."$3"."$4}' | sort

Get all unique IPs connected to your web server:

netstat -nat | sed -n -e '/ESTABLISHED/p' | awk '{print $5}' | sed 's/\./ /g' | awk '{print $1"."$2"."$3"."$4}' | sort | uniq -c | sort -n

On Linux
Get all IPs connected to your web server:

netstat -ntu | sed -e 's/::ffff://g' | awk '{print $5}' | cut -d : -f1 | sort -n

Get all unique IPs connected to your web server:

netstat -ntu | sed -e 's/::ffff://g' | awk '{print $5}' | cut -d : -f1 | sort | uniq -c | sort -n

RIP RJD

2010-05-16T14:38:00.001-07:00

The Last in Line

The mysterious Data Center Technical Specialist certification

2010-03-22T10:39:00.000-07:00

On March 4, I received an email from Novell Technical Training that I had received a Novell certification for "Data Center Technical Specialist". This came as a surprise to me because I had not applied for this certification, taken any tests for this certification, not had I even heardof this certification.

Due to a cross marketing agreement with the Linux Professional Institute, I had applied for and received the Novell Certified Linux Administrator certification a few weeks prior. This seemed legitimate to me, as I had extensive experience with SUSE Linux and my Linux skills are still sharp. I continue to perform Linux server administration as part of my daily work.

However, I am not quite sure what the Data Center Technical Specialist is supposed to represent. Confused, I wrote to Novell Training asking what the certification meant. I received this equally mysterious reply:

Thank you for contacting Novell Training Services. You have received the certification as part of some changes we have made recently to our partner requirements. As part if these changes, some of the exams/certifications you have now count toward the new certification.

I searched the official Novell Certification web site, and this certification does not appear anywhere. I suspect, but can't confirm that it may be part of the Solution Provider program. As such, it is probably more of a value to Novell sales than to an individual technician. I remain somewhat baffled.

Fuzzy string matching in PostgreSQL

2010-03-03T09:59:00.000-08:00

A recent project required me to use fuzzy string matching, or sound alike matching, in an application that searched a list of names. It turns out there is a contrib module for the PostgreSQL database called fuzzystrmatch that provides several different matching algorithms.

The task at hand involved rewriting a legacy application, originally in PICK, in Ruby on Rails. The PICK application used a soundex search to find names of people that sounded like the search string.

Three algorithms are available as PostgreSQL functions (after installation of the fuzzystrmatch module). They are soundex(), levenshtein(), and metaphone().

Both soundex and metaphone convert a string into character codes. Soundex uses 4 characters and metaphone uses a configurable number of characters. Levenshtein directly compares two strings and returns an integer indicating how well the two strings match.

After some trial and error, I found that metaphone produced better results than soundex. I didn't test the Levenshtein function.

To improve the results, I added a classic substring search using ILIKE. The combination of ILIKE and metaphone gave me a broad, but reasonably accurate fuzzy string search.

Linux: fuser to find processes on TCP ports

2010-02-26T13:39:00.000-08:00

Note: This is for Linux only. The Mac (BSD) version of fuser does not handle TCP/UDP ports.

Once or twice a year, I run into a problem where a process is using a TCP port and I need to find out which one. I am documenting it here for the next time so I don't have to look it up in man pages.

To see all processes, run fuser as root or with sudo.

To list all processes connected to TCP port 22:

fuser -n tcp 22

Rails 2.x scaffolding field types

2010-01-21T14:39:00.000-08:00

Dude, where's my CRUD?

One of the powerful features of Rails 1.x was the ability to generate CReate, Update, and Delete (CRUD) admin screens automatically using the scaffolding script built into Rails.

The original scaffolding read the database models and created basic, but usable screens to let you add and edit database records. When the 2.x release of Rails came out, scaffolding lost that power. Now, you have to manually specify each table field and type on the command line when running scaffold. If you don't, the generated screens will be empty.

Worse, a basic reference to all valid field types was missing. Here are all the valid types I have been able to dig up:

string
text (long text, up to 64k, often used for text areas)
datetime
date
integer
binary
boolean
float
decimal (for financial data)
time
timestamp

A mapping of the scaffolding types to data types in corresponding databases can be found on Overooped.

Here is an example of using 2.x scaffolding with data types, run from the Rails application root directory:


ruby script/generate scaffold Modelname name:string title:string employed_on:date remarks:text

Here is an example of using rails 3.x scaffolding with data types, run from the Rails application root directory:


ruby script/rails generate scaffold Modelname name:string title:string employed_on:date remarks:text

You ARE your operating system (among other things)

2009-12-24T13:32:00.000-08:00

Whenever you make a choice among products with similar functions, that choice spills over into the realm of social status.

Cars are a common example where social rank often goes with brand, and even within brand, by model, and even within model, by variations, upgrades, and badges. All signify some social status. I became acutely aware of this after purchasing a car that did not fit my image. It was uncomfortable for everyone.

Your operating system

You can see the social aspect tied to an operating system by looking at the Apple "I'm a Mac" campaign, and the weak Microsoft "I'm a PC" campaign response. A choice to run Linux or other operating system also carries connotations and shared group identity. In this sense, you are your operating system.

Your collection of choices

Whether conscious or not, decisions and choices about purchases you make, where choices are available, weave together part of your social tapestry. The schools you attend, where you work, your clothes, your car, where you live, and yes, your operating system. You are those things, at least socially.

The question is, in the context of a consumer society, can a choice be made without the attachment of social status?

Calculating the square root of 2 longhand

2009-12-05T14:13:00.001-08:00

There are many numerical methods to calculate square roots. This is the long hand method I learned in junior high. It produces one (accurate) digit at a time, but the working numbers get larger each iteration. Eventually, it bogs down because it is doing math with integers hundreds, then thousands of digits long.

Still, it is fun for tinkering. If you run this Ruby script as is, it will calculate the square root of 2 to 1,000 digits. To adjust the precision, change the number of iterations on this line:

while iterations < 1000

Here is the entire script...


#!/usr/bin/ruby
# calculate a square root of 2 using longhand

def newroot(divisor, doubleroot)
  # calculate new root
  # formula is doubleroot _ * _ = closest to divisor
  # try 9, then 8, then 7 ...

  i = 9
  while i >= 0
    multiple1 = (doubleroot.to_s + i.to_s).to_i
    multiple2 = i
    product = multiple1 * multiple2

    if product <= divisor
      # found new root
      root = i
      # get modulus
      modulus = divisor - product
      break
    end
    i = i - 1
  end


  return root, modulus
end

roots = Array.new

# the nearest root to 2 is 1

root = 1
roots.push(root)
remainder = 2 - root

doubleroot = root * 2
divisor = remainder * 100

iterations = 0

while iterations < 1000
  root,remainder = newroot(divisor, doubleroot)
  roots.push(root)

  # compute new doubleroot 

  return root, modulus
end

roots = Array.new

# the nearest root to 2 is 1

root = 1
roots.push(root)
remainder = 2 - root

doubleroot = root * 2
divisor = remainder * 100

iterations = 0

while iterations < 1000
  root,remainder = newroot(divisor, doubleroot)
  roots.push(root)

  # compute new doubleroot
  doubleroot = roots.to_s.to_i * 2

  # compute divisor
  divisor = remainder * 100

  iterations = iterations + 1

  print "iteration " + iterations.to_s + "\n"
end

print "final roots are " + roots.to_s + "\n"

Root 2: 500 million glyphs

2009-11-10T08:56:00.000-08:00

Having generated a text file with the first billion digits of the square root of 2, I started thinking about how to convert it to text.

First Try

My first try was an attempt to convert groups of 2 or 3 digits into a character using the ASCII code table. This was not very straight forward for several reasons. To start, many characters in the range from 0-255 are either non-printable or produce symbols or punctuation. To get around this, I decided to only use the part of the table that started with the numbers and went through the end of the lower case letters.

That left some 2 digits numbers that had to be scaled up into the useful range and many 3 digit numbers (everything greater than 255) had to be scaled down to the range. I did get a script working that accomplished this, but the results were not satisfying.

Second Try

I decided to use a simpler solution and convert each pair of digits into a letter using the range 1-26. To do this, I added one to the modulus of 26 and each digit pair and indexed it into the alphabet. Because 26 does not go into 99 evenly, this produces a slight bias against the last few letters of the alphabet but I was will to live with the result.

As an example, the first two digits are 14. 14 divided by 26 is 0 with a modulus (remainder) of 14. Adding one to 14 is 15 returning the letter O. The next two digits are 14 also returning O. The next two are 21 returning the letter V.

At the end of another scriptaculous ruby adventure, I had converted my billion digit file into 500 million letters.

Signals in the noise

To make the file easy to search, I wrote the resulting text file out in 72 character lines. Yes, this causes some loss of continuity at line breaks, but again, I was willing to live with it.

I quickly found the string with my name "KEITH" 65 times. The first occurrence was 7,398,524 digits into the square root of 2. My wife's name appeared 125 times.

I found "GOD" 33,663 times with the first appearance at 3,186 digits.

As expected, the smaller the text string, the more likely it is to be found. Most words and strings up to 6 characters can be found in the first billion digits of the square root of 2. Strings 7 characters and longer, like "CALIFORNIA" are often not found.

For fun, I ran a few searches for dates and numbers in the integer file, finding things like my birthday and dates of historic events. Numeric strings are much easier to find.

Next, I'll post one of my ruby scripts to calculate the square root of 2 digit by digit. It is very slow and inefficient, but fun for tinkering. I have a few more ideas for grappling with the root of 2.

Root 2: the shallow end of infinity

2009-10-23T12:03:00.000-07:00

A couple of months ago, I got the semi-psychotic idea to search for numbers and strings in the digits of the square root of 2. The idea is hardly unique. There has been a fair amount of research into this area, a variation on the infinite monkeys typing on a typewriter theorem.

Why Root 2?

Because it wasn't Pi. Pi is the probably the most studied math constant, so I wanted to start with something fundamental that wasn't pi.

I needed an irrational constant, one that generated an infinite stream of random numbers, and root 2 was as good a choice as any. I might argue that root 2 is even more commonly encountered than pi, in that it is found in the hypotenuse of a 1x1 square. Surely, 1x1 squares are more common than circles in the modern world.

Infinite Integers

The first challenge was to find or generate the integers of root 2. With a quick search, I found up to 10 million digit files that had been created by NASA. While a good start, I wanted a lot more than 10 million digits. I ended up using the 1 million file from NASA as a control file to make sure that any other method I used to generate digits was validated by NASA. My starting target was 1 billion digits, but I wanted the ability to go to an arbitrary number.

When I started thinking about how to generate digits with a computer, I realized there were some unique hurdles in trying to calculate infinite precision numbers. First, standard machine data types fail at the task. 64-bit integers give you a large work space, but floating point numbers are notorious about losing precision. The math libraries of most languages work to about 15 digits, so those were out.

I found more than 10 methods documented to compute square roots.

Baby Steps

The first attempt to calculate the square root of 2 was actually written by a colleague who was intrigued enough to implement the Babylonian method in clojure. The first few tests seemed promising and matched the NASA file up to roughly 25,000 digits. Where it broke down was after about 20 iterations where the run time exceeded 18 hours. Getting to a very large number of digits did not appear to be practical.

The next attempt was one I wrote using the Duplex method (in ruby). Duplex has the advantage of computing one digit at a time and I was hoping this would avoid some of the problems with doing multiplication with millions of digits. There may have been a flaw in my ruby implementation because it got off track past digit 53 when I compared the results with the NASA file. I was careful to follow the example I found in Wikipedia, but I must have made some kind of mistake because it always diverged.

Standing on Shoulders

Like any good programmer, I quickly decided to search for working code that someone else had written. I found a very old C++ implementation, but also a handful of working programs. The one I tried first was PiFast.

PiFast is a Windows program that can calculate many constants in addition to Pi. Since my main computer is a Macbook, I loaded it into my VMware virtual machine running XP. As you might imagine, it was more than a little CPU intensive. After a few short trial runs, I let it loose on a 1 billion digit computation that for all intents locked my machine for about 11 hours. At the end, I had a file of 1 billion digits, the first million of which validated perfectly against the NASA file.

To validate and test the file -- a 1 gigabyte file -- I wrote a number of scripts (ruby is my scripting language of choice these days) to test the digits, and manipulate the file in several ways.

In my next post, I'll talk about how I converted digits to letters and what my early searches turned up.

Introduction to AutoFS in Mac OS X

2009-09-25T12:17:00.000-07:00

Note: I originally published this article on Low End Mac

OS X uses an AutoFS code stack based on Sun's Solaris version of Unix. Many of the advanced features are not documented very well, and this can be an issue unless you are familiar with Solaris. I was not and had to do quite a bit of digging.

AutoFS is often used in enterprise environments to set up network-based home directories and other network mounts for users at login. It can also dynamically mount network shares on access.

OS X auto_master and auto_home

The /etc/auto_master file controls the auto-mounted Network File System (NFS) file systems. If you are going to mount NFS volumes from a Linux server, there is one gotcha that I covered in an earlier blog post.

The auto_master defines all "maps" which are collections of automounts related by mount point and organized in one file (or directory service entry). Here is what the default file looks like on my Mac:

#
# Automounter master map
#
+auto_master            # Use directory service
/net                    -hosts          -nobrowse,nosuid
/home                   auto_home       -nobrowse
/Network/Servers        -fstab
/-                      -static

The plus (+) sign in front of the auto_master entry tells OS X to look in the directory service (Open Directory, LDAP, etc.) for an automount record and use it if found.

Notice the /home entry is set to auto_home, and because it is not a full path, it is assumed to be /etc/auto_home. It is an example of an indirect map. The mount point in the local directory is defined, but the remote mounts are defined in the /etc/auto_home map file. Network users who login to the local machine will have their home directories mounted in /home according to the details in /etc/auto_home.

Here is the default /etc/auto_home file:

#
# Automounter map for /home
#
+auto_home      # Use directory service

Once again, we see the plus sign telling OS X to look for an auto_home record in the directory service. No further details are defined.

The last two lines in auto_master handle NFS mounts defined in the /etc/fstab file, the common file system mount table in Linux and other Unix flavors. The /etc/fstab file is deprecated in OS X and not recommended.

Applying changes to autofs

The automount process will not detect changes made to auto_master or other map files unless you tell it. This command tells the process to read all map files again:

sudo automount -vc

AutoFS wildcards

Wildcards can be used in mount map files to allow directory substitution. For example, if you had this defined in auto_master:

/opt auto_public

And this defined in /etc/auto_public:


*       nfs.mydomain.com:/public/&

Then, when /opt/bin was accessed, nfs.mydomain.com:/public/bin would be mounted on /opt/bin. The same would apply for any subdirectory accessed under /opt.

Other Map Types

OS X AutoFS supports direct maps, where the local mount points are defined inside the mount map file, and indirect maps, where the local mount point is defined in auto_master. The wildcard example above is an indirect map. There are also executable maps where the mount map file is actually an executable shell script that returns the names of the mount points within the trigger folder. Exploring executable maps is left as an exercise for the reader. Finally, you can define static maps in /etc/fstab or in the Directory Utility Mounts tab.

Other file system types

All of the examples shown use the NFS file system. OS X autofs can also handle Apple File System (AFP) and Microsoft Server Message Block (SMB) file systems.

To use these file systems, add the -fstype=afp and -fstype=smbfs options when defining the remote mount points. (Note: You cannot use smbfs for remote home directories unless you are using the Microsoft Active Directory service plugin.)

Playing nice with Linux NFS

2009-06-27T09:10:00.000-07:00

I naively assumed that mounting an NFS volume exported by Linux would be uneventful, and it should be. My initial attempt to manually perform an NFS mount failed without any client side error message.

Checking system logs on the Mac revealed nothing.

The Linux NFS server (Red Hat Enterprise 5) complained with this warning:


nfsd: request from insecure port (192.168.7.130:49232)!

After some Internet sleuthing, I found Mac client side NFS tries to mount NFS volumes over high TCP ports (>1024). You must explicitly tell the Linux NFS server to accept mount requests on high ports by adding the "insecure" option to /etc/exports. For example,


/nfstest 192.168.206.0/24(rw,async,insecure)

Then, NFS mounts from OS X should work as expected.

Automating FTP on the Mac

2009-04-12T09:54:00.000-07:00

There is no shortage of GUI FTP programs, but kicking it old school on
the command line allows you to easily automate uploads and downloads.
The best part is, there is nothing to install. Everything you need
waits patiently behind the warm glow of a Terminal session.

The Mac command line FTP program
The default command line FTP program in OS X 10.5 resides at:
/usr/bin/ftp

By all outward appearances and behavior, the Mac FTP program is
the standard BSD version. The man page is the standard BSD
page and contains a wealth of useful information.
A typical command line FTP session is interactive and goes something
like this:

issue commands (ls (list), get (download), put (upload))

quit

If you have a repetitive FTP task, the fun quickly fades into a
mind numbing exercise. This is where FTP automation
shines.

The magical .netrc file
What makes FTP automation possible is a magical, little known file
called .netrc. The .netrc file is a plain text file that is
hidden (the file name starts with a period) and lives in the root of
your home directory. The .netrc file allows FTP to perform automatic
logins to FTP servers based on the name.

The .netrc is not created by default. You have to create it manually.
To create an empty .netrc file, open a Terminal and use the following
commands:


touch .netrc
chmod 700 .netrc

It is critical that you issue the chmod command to
set the permissions so that only the owner of the file can view it. If
the permissions are not set correctly, the FTP client will assume it has
been compromised and will refuse to use it.

Inside the .netrc, you define a block of settings for each FTP
server you use, including the machine name, the login ID and the
password. Here is what a typical block for a mythical FTP server:


machine myftpserver.com
  login myuser
  password mypassword

There are additional settings that can be included. Check the FTP man
page for more. You can test your settings by typing "ftp
myftpserver.com" at a Terminal prompt and it should automatically login.
Note that you can store multiple FTP server logins in the .netrc file.

Sending FTP commands from a BASH shell script
Once logins are automated, the final piece of the puzzle
is to script a set of FTP commands. The following example uses an
advanced BASH shell scripting technique called a "here" document
to group the FTP commands to be sent to the server.


#!/bin/bash
/usr/bin/ftp -d myftpserver.com << ftpEOF
   prompt
   put "*.html"
   quit
ftpEOF

The FTP command is issued with the -d flag (debug mode) to make it more
verbose. That makes any kind of error more obvious. The connection is
made to myftpserver.com using the ID and password from the .netrc file.
Once the connection is made, the rest of the commands are issued one at
a time until the end of the "here" document at the second "ftpEOF". Note
that any valid FTP commands can be sent. In the example, the
prompt command tells FTP not to prompt for multiple file
operations, then the put uploads all files with an html
extension. If you want to go the extra mile, you can extend the shell
script and do things like reconnect to the FTP server to verify the file
sizes of your uploads.

While there are several ways you can automate FTP, the nice thing about
this method is that it is portable to Linux or any other Unix system.

system_profiler - gathering hardware/software configuration

2009-03-21T08:19:00.000-07:00

Another useful command line tool, system_profiler reports detailed information about the hardware and software configuration of the Mac.

It reports the same information you can find in the Apple menu / About This Mac / More Info dialog.

One of the things that makes it so useful is that you can look at the configuration of a system you are connected to remotely, like through a secure shell connection.

There are three details levels available:

mini - report with no personal information
basic - basic hardware and network information
full - all available information

You select which detail level by passing it using the -detailLevel switch. For example:

system_profiler -detailLevel mini

If no detail level is passed, the default level is full.

Because it generates so much information, you typically pass the output to the "less" utility, redirect it to a text file for browsing, or filter out what you are looking for using "grep".

Enable/disable Fast User Switching from the command line

2009-02-25T10:18:00.000-08:00

To enable fast user switching, use the defaults command from the terminal:


defaults write /Library/Preferences/.GlobalPreferences  MultipleSessionEnabled -bool YES

To disable fast user switching:


defaults write /Library/Preferences/.GlobalPreferences  MultipleSessionEnabled -bool NO

Apple System Logger

2008-12-27T06:08:00.000-08:00

Despite appearances, the system logging facility is unique on OS X 10.4 and newer. The syslogd daemon used on most Unix-like operating systems was replaced with the Apple System Logger (ASL).

Instead of using a new program name, Apple just used the traditional Unix system logger name. I think this can be a source of confusion and was not a great decision.

Binary Logs, then Text Logs

Instead of logging everything to text log files in /var/log, ASL logs all messages to a binary database file, /var/log/asl.db. From there, certain messages are written to the more familiar text based logs based on settings in /etc/syslog.conf.

In a way, Apple hacked the syslogd process, but did so in a way that maintains some level of backward compatibility with the knowledge and tools of system administrators coming from other Unix-like systems. I admit I was quite surprised to find out how this mechanism has been implemented by Apple.

The binary asl.db database is cleaned periodically, removing older messages from the database and keeping it from growing indefinately. The text based logs are still compressed and rotated as usual.

Reading log files

You can use the GUI utility, Console.app, to browse the Apple binary log database. It has a nice case insensitive search ability.

The command line interface to the Apple binary log is syslog.

Entering syslog with no parameters dumps the entire database. You can pipe the output through a grep command looking for a text string. For example,


syslog  | grep Error

returned a large number of errors showing someone trying to login to my system via SSH as root:


Sat Dec 20 16:18:42 white com.apple.SecurityServer[22] : checkpw() returned -2; failed to authenticate user root (uid 0).

Yikes! Not to worry, when remote logins (SSH) are enabled, the default configuration does not allow root logins. According to the man page, you can also search for particular keys in the database using the -k option. Unfortunately, the list of valid keys is not in the man page.

Logging from a script

Finally, you can send a message to the log file using the -s parameter. Messages in the traditional syslog system are categorized by severity, or log level:

Emergency (level 0)
Alert (level 1)
Critical (level 2)
Error (level 3)
Warning (level 4)
Notice (level 5)
Info (level 6)
Debug (level 7)

To send a Warning message, use:


syslog -s -l 4 "this is a warning"

produces this in the log:


Sat Dec 20 18:42:29 white syslog[40323] : this is a warning

Disabling hardware via kernel extensions

2008-12-22T01:24:00.000-08:00

You can disable hardware components in OS X by deleting (or renaming) the kernel extension directory that contains the driver for that piece of hardware.

Kernel extensions are stored in subdirectories in /System/Library/Extensions.

In some cases, there is more than one driver involved, so you need to delete or rename multiple directories.

Drivers for AirPort wireless:

AppleAirPort.kext
AppleAirPort2.kext
AppleAirPortFW.kext

Drivers for Bluetooth wireless:

IOBluetoothFamily.kext
IOBluetoothHIDDriver.kext

Drivers for external mass storage (external hard disks or USB keys):

IOUSBMassStorageClass.kext
IOFireWireSerialBusProtocolTransport.kext

You may have delete or rename the directories again after a system update because an update may restore the drivers. After removing the drivers, reboot to make sure the hardware is disabled.

Another way to enable and disable root

2008-12-19T11:29:00.000-08:00

I stumbled across another way to enable and disable root. There is a program called dsenableroot that can be used to both enable and disable the root account.

The full path is /usr/sbin/dsenableroot and you must be an admin level user to run it. In both cases, you will be prompted for your own user password. When root is enabled, you are prompted to set the root password.

This sequence enables root:


 dsenableroot
username = keithw
user password:
root password:
verify root password:

dsenableroot:: ***Successfully enabled root user.

This sequence disables root:


dsenableroot -d
username = keithw
user password:

dsenableroot:: ***Successfully disabled root user.

I generally like to have root enabled. Here is my older post on enabling root.

Power settings from the command line

2008-12-16T16:24:00.000-08:00

By request, I did a little digging to find out how to view and set power settings from the command line.

The command to use is pmset.

Instead of inventing the wheel again, please see this most excellent article at University of Utah.

Speeding MacBook 802.11g connections

2008-12-16T16:16:00.000-08:00

My wife got a new MacBook a few weeks ago and was complaining of slow 802.11g wireless speeds at home. Normal troubleshooting went nowhere. Configuring the access point a little differently improved the situation.

My wife took the MacBook into an Apple store and diagnostics were run on it showing no connection or performance issues. At home, it was still slow. This was odd because I have a slightly older MacBook and an even older Linux laptop working fine on the same wireless network.

I searched the Interwebs and found a post on an Apple related forum, forget where now, that suggested Macs liked Channel 11 on 802.11g networks.

With nothing to lose, I logged into the LinkSys access point and changed it to listen on Channel 11 (2.462 Ghz) by default. After this change, the new MacBook wireless performance was much better. I have no idea if there is some affinity for Channel 11 in Apple's hardware or software stack, or if there is less interference on that frequency, or if there is some other explanation. In any case, I thought it was worth sharing.

Intro to ipfw

2008-12-16T15:48:00.000-08:00

The low level firewall built into FreeBSD, ipfw, is also a part of OS X. It is one of the juicy UNIX bits that came along for the ride. The program itself lives at /sbin/ipfw and is a stateful packet filter capable of controlling access to and from specific IP addresses, ports, or combinations of each.

The Firewall in System Preferences

The firewall settings in Preferences are application level, and are separate from the low level rules set by ipfw.

On Leopard, you can get to the Firewall settings by going into System Preferences, select the Security icon, then the Firewall tab. The default setting is to allow all incoming connections. This is a little misleading because OS X comes with network services generally locked down until you enable them.

If you enable sharing, for example web sharing via the Apache 2 server in the Sharing Preferences, you will see Web Sharing appear in the firewall box in System Preferences. For most people, this level of control is fine, especially if you have another firewall between you and the Internet. For advanced configurations, ipfw can meet your needs.

Listing ipfw rules

To list all active ipfw rules:
sudo ipfw list

Each rule is assigned a number up to 65535. Lower numbered rules are executed first, and if they don't apply to the packet in question, it moves down the list until it matches a rule. It is possible for a packet to be injected back into the rules, but usually a packet matches one rule and is either accepted, denied, or routed to a new destination.

Clearing all ipfw rules

To list all active ipfw rules:
sudo ipfw flush

Creating custom rules

The command line interface to ipfw is somewhat arcane, and you can get yourself into trouble if you don't know exactly what ports you need open and why. Here is an article that has some sample configurations and command line syntax examples.

While I normally prefer the command line to GUI tools, if you are just starting to use ipfw, I recommend the very friendly NoobProof, a program that offers a useful GUI front end to ipfw.

NoobProof comes preconfigured with all the common Mac services that you can enable or disable at the network level using ipfw. It also has an install tool to create a script that saves your configuration and starts it automatically after a reboot. It is a truly wonderful tool and a much safer way to get started with ipfw.

Airport command line utility

2008-12-16T15:46:00.001-08:00

OS X Daily reports on a hidden but useful command line utility to monitor and tweak wireless connections called airport.

Date validation in Ruby using the Date object

2008-12-16T14:28:00.000-08:00

Ruby has a date library to help you manage and manipulate dates. The date library provides Date and Date/Time objects that can be used to validate dates. In addition to straight Ruby, this helps work around a particularly nasty problem in version 1.x scaffolding date fields created by Rails.

The Date object

To use the date library, add this line near the top of your Ruby program:

require 'date'

The date library provide the Date object and the DateTime object, which adds time attributes to the Date object. To create a new Date object, pass the year, month, and day to the new Date constructor:

mydate = Date.new(2008, 7, 11)

To get the current date, use the Date.now method. You can access different attributes of a date object like this:

mydate.year mydate.month mydate.day

Converting a date object to a string returns a "YYYY-MM-DD" format. For example:

mydate.to_s "2008-07-11"

A simple validate function

One way to test for a valid date is to try to create a Date object. If the object is created, the date is valid, and if not, the date is invalid. Here is a function that accepts year, month, and day, then returns true if the date is valid and false if the date is invalid.

  def test_date(yyyy, mm, dd)
   begin
     @mydate = Date.new(yyyy, mm, dd)
     return true
   rescue ArgumentError
     return false
   end
 end

The creation of the date object is wrapped in a begin...rescue...end block so that the error can be trapped if the date is invalid. Ruby throws an ArgumentError if the date is invalid and in that case, the function returns false. This version uses an instance variable (@mydate) because I wrote it to be used in a Rails application.

The nasty date problem in 1.x Rails scaffolding

In version 1.x of Rails scaffolding, date fields in the edit view are generated with three drop down boxes, one for the year, one for the month, and one for the day. However, you are free to select invalid month/day combinations like February 30 and November 31. When these parameters are passed to the Rails update function in a scaffolding generated controller, it fails.

The crude solution I used to work around it was to edit the date prior to the attempt to save the record. In this case, the view sends the year to the controller in the dob(1i) field, the month in dob(2i), and the day in dob(3i). I convert them to integers, then call the function above to validate the date. If the date is invalid, I set a flash message and return to the edit screen.

    # date validation code
   @allparms = params[:dependent]
   @yyyy = @allparms["dob(1i)"].to_i
   @mm = @allparms["dob(2i)"].to_i
   @dd = @allparms["dob(3i)"].to_i

   if test_date(@yyyy, @mm, @dd) == false
     flash[:error] = "Invalid date " + @mm.to_s + "/" + @dd.to_s + "/" + @yyyy.to_s
     redirect_to :action => 'edit', :id => params[:id]
     return
   end

It seems like this rather basic problem would have been worked out in the Rails scaffolding, but it wasn't. I haven't tested Rails 2.0 scaffolding to see if it has similar issues. Other people have come up with more comprehensive solutions, like the Rails date kit. Take a look at something like that if you want more than a quick and dirty solution.

Beginning a Rails app -- for beginners

2008-12-16T14:25:00.000-08:00

After some trial and error, I got my first significant Ruby on Rails application off the ground. By significant, I mean a dozen related tables, about 20 controllers, and some tricky business logic. I had been exploring Rails for a while and created a few toy applications, but hammering this one out still presented a challenge.

If you are used to programming CGI, PHP, or any application not built on a Model-View-Controller (MVC) framework, the learning curve is steep. While I can't offer up best practices -- I still have too much to learn -- I can offer up how-the-hell-do-I-get-started practices. For my own benefit, I created a list of steps to ease the pain of getting started.

Don't start here

As I mentioned, unless you are coming from another MVC framework, the learning curve is kind of steep. Rails has lots of nooks and crannies, lots of helpers and shortcuts that are not obvious at first. So, even though this is a beginning Rails article, it is not a good place to start learning about Rails. I suggest starting with a good book.

The three books I always have at hand when working in Rails are:

Programming Ruby: The Pragmatic Programmers' Guide (pickaxe)
Agile Web Development with Rails
Rails Cookbook

Also, I spent no time telling you how to install or set up Rails on your system. There are many places to find such information, and if you are running a Mac, you already have Rails installed, though you may need to set up your database.

Four terminals at Rails central

I do my text editing with vim (and sometimes nano), performing all work in the Terminal. Textmate and other GUI editors are popular but I prefer to stay in the Terminal. I have found that 4 terminal windows (or tabs) is ideal for working on a Rails application. That's 4 terminals for each Rails application, so if I was going to work on two at the same time, (unlikely), I would open 8.

I keep the Terminals ordered left to right and dedicate the first to working on Models, the next one for Views, the next one for Controllers, and the fourth one for performing other tasks such as debugging, reviewing logs, running scripts, etc. Keeping them in MVC order helps me stay organized while working and lets me switch between related views and controllers quickly.

Of course, I also have a browser window open to test changes as I go.

Start here

Keep in mind, these are not rules, just basic guidelines that can help you approach a new project.

create the app structure with "rails appname", this command generates the directory structure for the application and all the configuration files
create the database and tables (outside of rails), this is not the "rails" way, using migrations, but I prefer to do create/modify my database outside of rails
configure /config/database.yml with database details
create models, one per database table in app/models/
add validate helpers and validate methods to models in app/models/
create initial controllers/views using scaffolding with "ruby script/generate scaffold Model-name"; run once for each table/model using the singular form of the table name for the controller. (The model name is capitalized and singular).
add business logic to a controller and set instance variables with data that will be needed in the view
customize the default template in /app/views/layouts/. Layouts are used to hold common HTML and embedded ruby such as headers and footers for your pages. Usually, there is one layout per controller.
fill in the views (often rhtml files), one for each method in the controller. Views and layouts control the presentation.
repeat each step above starting with step 6 (the scaffolding step) for each table/model in your application, you can come back and add or change models later
create new controllers not related to any models with "ruby script/generate controller Controller-name" and add methods and views as needed
edit config/routes.rb to set the default URL for the application, rename public/index.html to index.html.old,
e.g. map.connect '', :controller => "main", :action => "list"
it must be the first map in the routes file.

For pure data entry forms, create an empty controller method with a corresponding view. The view will contain the form and call a different method to accept the data and display the results (in yet another view).

If authentication is needed, use a "before_filter" at the top of each controller to handle logins and set session data.

In Rails version prior to 2.0, the scaffold script generates non-RESTful code, while in 2.0 and later, it generates RESTful code. This is a concern if you were used to the way Rails worked before 2.0. I am not sure RESTful is really better, but either way it can be confusing if you upgrade to Rails 2.0 in the middle of developing an application and find that scaffolding no longer works the same way.

Rails is flexible

One of nice things about Rails is that it is flexible. You can use Rails helpers when constructing your views and web forms, or you can code them using standard HTML. If I was not comfortable using a particular form helper or could not get the syntax just right, I fell back to coding it in regular HTML. Rails had no problems merging form helpers and regular HTML in the same form within a view. In many parts of Rails, it is not all or nothing. You can use all, some, or none of the shortcuts.

The ease of modifying applications is another strength of Rails. It is relatively easy to add new functionality, tables, or features. You don't have to figure out everything before you start. It is also easy to rewire your application if you don't like the way it flows between screens or the way the pages are linked together.

Testing

I think the official best practice to test a Rails app is to write test scripts that assert conditions and report problems found. I haven't used test scripts yet, my skills are not advanced enough. Up to now, I have relied on three other techniques.

The first is using the "script/debugger" to stop my application at certain points so I can display the values of objects and variables. I think the script/debugger was either removed or replaced in Rails 2.0.

The quick and dirty way to test is to write data to :flash and view the contents of variables that way. The flash is a built in hash, part of the session data, used to display informational or error messages to the user. It is an easy way to debug simple problems.

For deeper problems, I write data to the log file using the logger object. It takes a little time to sift through all the stuff in the development log file, but you can see trends and what is passing through the application before and after the trouble spot.

Railsbrain

Finally, I want to point out an extremely useful reference web site for working with Rails: Railsbrain. It has an Ajaxy API reference for multiple versions of Rails along with syntax and sample code.

I expect to come back to this article in six months and laugh at the inefficiency of the way I started doing Rails, but for now, it was enough to get that first useful application done.

Ruby Snippets

2008-12-16T10:35:00.000-08:00

As I get deeper into the Ruby world, it is helpful to have a few snippets of working code and syntax for common tasks to copy as needed. I expect to add to this tiny reference over time.

updated July 9, 2008 to add a one liner to remove DOS line endings from a text file.

Formatting numbers

To format numbers, use the sprintf method.

x = 1.00
puts "x is $" + sprintf("%#0.2f", x)
puts "x is " + sprintf("%#2d", x)

The output is:

x is $1.00
x is  1

The defined? method

The defined? method can be used on any object to see if it exists.

x = 1
if defined? x
 puts "x is defined"
end

Case statements

Case statements are a more elegant way to handle a large number of conditions than using nested if statements.

x = 3
case x
when "1"
 result = "x is 1"
when "2"
 result = "x is 2"
when "3"
 result = "x is 3"
else
 result = "x is something else"
end

Substrings

Ruby has (at least) two ways to reference parts of a string object. One is to use the slice method:

x = "abcde"
y = x.slice(0,1)

The result is y = "a".

x = "abcde"
y = x.slice(-1,1)

The result is y = "e".

x = "abcde"
y = x.slice(2,3)

The result is y = "cde".

The second way is to treat the string object as an array of characters like the C programming language.

x = "abcde"
y = x[2,3]

The result is y = "cde".

One liner to edit a file in place

These one liners are also in the general Ruby article.

This command edits a file in place, performing a global text search/replace. The -p switch tells ruby to place your code in the loop while gets; ...; print; end. The -i tells ruby to edit files in place, and -e indicates that a one line program follows.

ruby -p -i -e '$_.gsub!(/248/,"949")' file.txt

One liner to remove DOS line endings from a file

A Unix or Mac text file uses a single line feed character to mark the end of a line (hex 0A). Files created on DOS or Windows use two characters, carriage return and line feed (hex 0D 0A). This command edits a file in place, removing the carriage returns.

ruby -p -i -e '$_.gsub!(/\x0D/,"")' file.txt

One liner to edit files in place and backup each one to *.old

ruby -p -i.old -e '$_.gsub!(/248/,"949")' *.txt

Boot Camp

2008-12-16T10:33:00.000-08:00

Boot Camp is a set of tools to make dual booting an Intel Mac with Windows XP or Vista easy. The main reason you might want to dual boot as opposed to running Windows in emulation is for the extra performance from running natively. You might need this for gaming or a demanding application like CAD or photoshop where running in a virtual is just too slow.

You must have a CD or DVD with the full version of Windows XP or Vista. An upgrade disk will not work.

Update firmware

Before beginning the process, Apple recommends updating the firmware on your Mac to the latest version. If you are applying all software updates, then you are probably already at the latest revision. If you are not sure, run the Software Update command from the Apple menu.

Boot Camp Assistant

The next step is to run the Boot Camp Assistant program in the Utilities application folder. The first button in the assistant prompts you to print the installation and setup guide. I highly recommend you take that advice. I didn't and (temporarily) ended up with a Mac that would not boot. The guide is 25 pages long and covers all the details to get Windows installed. For that reason, I'll just cover the juicy bits and tell you the mistakes I made along the way.

The assistant provides a disk partitioning tool that lets you carve out part of your disk for Windows. I allocated 12 GB and proceeded to installation.

Installing Windows

I don't own Vista, so the rest of this article applies to Windows XP Pro SP2.

The installation of Windows works exactly like it does on any PC. When you get to the part where you have to choose what partition to install Windows on, select the one labeled [BOOTCAMP]. Also, it is important to have Windows format the partition (as either NTFS or FAT32), or it will not be able to boot back into the partition after copying its files to disk. The mistake I made was not formatting the disk. Even though the Windows installer could copy files to the disk, it was not able to boot back into the partition, leaving my Mac temporarily out of order.

I was able to get back into OS X by holding down the mouse button during boot, which ejected the Windows CD. Then I printed the installation guide, where I learned about the formatting requirement.

Installing Windows drivers

After Windows is installed, the next step is to install all the drivers for the Apple hardware. This was one of the best thought out parts of the process shows Apple's superior focus on human interface design. All you have to do is insert the OS X DVD while in Windows and it will start the driver installation wizard. Apple includes drivers for the video, audio, built-in camera, wireless airport, bluetooth, and more. It was the most painless Windows installation I have ever done.

Booting between OS X and Windows

To boot into OS X from Windows, you have two options. The first is to right click on the Boot Camp tray icon that gets installed during the driver procedure and select the option to boot back into OS X. You can also choose to make OS X the default boot option in the Windows Boot Camp control panel applet.

To boot into Windows from OS X, go to System Preferences / Start Up Disk and select the Windows partition as the default.

Finally, you can select which operating system to start at boot time by holding down the Option button.

Keyboard quirks on a Macbook

The differences in Apple and Windows keyboards is a source of minor annoyance. The keys are mapped in a reasonably intelligent way, but there are enough differences between the systems that some Windows features are awkard to use.

For example, to perform a right click, you hold two fingers on the trackpad and click the mouse button -- not a natural maneuver. The installation guide has a table that shows which keys and combinations map to Windows equivalents. Fortunately, most applications have some ability to map keys to make using them more comfortable.

Getting rid of Windows

Finally, the Boot Camp Assistant program has an option that lets you delete the Windows partition and restore the space set aside for use with OS X again. So, experimenting with a dual boot set up is not a one way ticket. Easy to try, easy to undo.

Ruby

2008-12-16T10:31:00.000-08:00

Ruby is a popular text processing language that has gained a lot of momentum over the last few years, mostly due to the rise of Ruby on Rails. It has a lot of the good things from Perl, but implements a fully object oriented model and syntax.

Ruby is part of the default install of OS X.

Working with Ruby gems

Gems are Ruby packages that can be installed to extend the language or provide additional functions. Gems are managed using the gem command. To find all Ruby gems (packages) currently installed:

gem list

To install a new gem:

gem install package

To update the gems package manager:

update_rubygems

Taint mode

For untrusted scripts, add -T flag to ruby at the start of the script:

#!/usr/bin/ruby -T

One liner to edit a file in place

One liner to remove DOS line endings from a file

One liner to edit files in place and backup each one to *.old

ruby -p -i.old -e '$_.gsub!(/248/,"949")' *.txt

Intel vs. PowerPC Macs

2008-12-16T10:30:00.002-08:00

I recently purchased a new Intel Macbook to replace my trusty iBook G4. Much has been written about the Apple's switch from PowerPC to Intel processors and there is no denying the market power of Intel, but I have some comments now that I have spent a little time with both kinds of Mac.

RISC vs. CISC

When comparing Intel and PowerPC micro processors, a common theme used to be the Intel complex instruction set (CISC) vs. the IBM reduced instruction set (RISC). This line of reasoning may have had some merit in the 1980s or early 1990s, but processor architecture has blurred, with each family incorporating the best designs of the other. I don't read much about this any more because it mostly doesn't apply.

Big endian vs. little endian

The endianness of a processor refers to the byte order of data in memory. Specifically, big endian processors, (traditionally the PowerPC line), stored the most significant bytes first, while little endian, (the Intel line), stored the significant bytes last. If you have ever debugged data or instructions in memory, big endian is the logical way you would expect to see things, a more natural or human way to read memory. Little endian data looks garbled to me.

I started my programming life long ago on an IBM mainframe (big endian) and wrote quite a few lines of assembly language. I actually enjoyed working with the mainframe register architecture and assembly was a joy. When I started working with Intel PCs, assembly language became a nightmare. The registers could be addressed partially using one mneumonic, fully with another, and the debugger showed bytes in the strange little endian order. It was a real buzz kill.

Today, with optimizing compilers and high level languages, it doesn't make much difference any more. Most modern languages are so high level (my current favorite is Ruby), that endianess is transparent. Some processors can even be programmed to change their endianess. So, other than personal preference, and a slight distrust of engineers that prefer little endian, this is a difference that doesn't carry much weight.

Power, heat, and batteries

PowerPC processors have nearly always been more power efficient than Intel. This translates into less heat (less time running a cooling fan), and longer battery life. The battery life between my new Intel Core 2 Duo Macbook and my G4 iBook is dramatic. Even though the G4 is over 2 years old, it still gets about an hour and half of extra battery life. Certainly, part of that is because it also has a smaller and dimmer screen, but the processor architecture is also a factor. I was shocked at how fast the Macbook battery ran out the first week I had it. It is still better than a Windows laptop, but a step backward compared to the G4.

Binaries

The operating system was not the only thing that had to be compiled to run on Intel processors. To take advantage of native speed, every application had to be recompiled. Some software vendors took their time building Intel binaries. Apple provided the "Universal binary" solution ahead of the switch so the application hit wasn't too harsh.

The virtual plus

Apart from the ability of Intel to supply Apple with enough processors, the switch enabled fast virtual machine technology that doesn't rely on software emulation to work. Applications like Parallels and Vmware Fusion give the Intel based Macs a vastly improved capability to run Windows, Linux, and other operating systems with OS X as the host. I tried running Linux on the free Q virtual machine on the G4 and it was almost too slow to use. Running VMware Fusion, I have a couple of Linux distributions running graphical X desktops at near native speed. In this sense, the switch to Intel was a virtual plus.

Does it matter?

How you weigh the positives and negatives above determines how you view the switch. In the end, the best things about a Mac still exist in the Intel world, and apart from the good folks at IBM, Motorola, and Intel, most people won't notice.

How to turn off Dashboard

2008-12-16T10:30:00.001-08:00

The Dashboard is a handy feature of OS X to keep information flakes and small tools at the ready.

To turn Dashboard off, run the following command from the Terminal:

defaults write com.apple.dashboard mcx-disabled -boolean yes

To turn Dashboard on:

defaults write com.apple.dashboard mcx-disabled -boolean no

Because Dashboard is a component of the Dock, you need to restart the Dock after making either change:

killall -1 Dock

Advanced BASH scripting guide

2008-12-16T10:28:00.000-08:00

User forbade pointed out this great guide from the Linux Documentation Project:

Advanced BASH scripting (pdf)

It is also available in other formats.

Amazon S3 from the command line

2008-12-16T10:25:00.002-08:00

Amazon.com offers a variety of web services to make building highly scalable web applications easier. One of those services is called the Simple Storage Service (S3). It provides a way to store and retrieve files (up to 5 GB each) on Amazon's distributed servers. S3 has both a SOAP and REST API. You can use the S3 REST API with only the native command line utilities installed by default in OS X Leopard (and Linux).

Why use Amazon S3?

Amazon S3 is designed to be robust, fast, and inexpensive. It is ideal for serving small images or small files for web sites. It is also a good solution for off site backups (less than 5 GB) because the storage is redundant and inexpensive. For very large backup files, it is not the best solution because of the file size limitation (which can be worked around in some cases), and because the bandwidth is prohibitive.

How does S3 work?

While S3 is cheap, it is not free. To use it, you first need to sign up with Amazon Web Services and obtain an access key ID and secret key. The access keys are required to authenticate to Amazon and to use the API.

Conceptually, S3 stores files (objects) in buckets. First, you create a bucket, then you store objects in the bucket. The combination of the bucket and object ID must be unique. To S3, all objects are binary blobs. You can attach some meta data to an object when it is stored, but S3 doesn't verify that the meta data accurately describes the object.

With the REST API, standard HTTP transactions are used to GET, PUT, and DELETE objects in buckets. There is no incremental update option. To update an object, upload and replace it with the new version.

Using S3 from BASH

As part of a consulting project, I was asked to upload rotating backup files to S3. The backup files consisted of compressed source code and database dumps.

I started by looking at the sample code on the Amazon developer site. I downloaded some working PHP code that used the REST API and began hacking on it. However, I ran into a serious problem when I tested uploading one of the backups. The sample code was designed for small files that could be uploaded in a single HTTP transaction. It had a limitation of 1 MB, while my backup files were around 250 MB. I could have tried to retool the code to break apart the backup file and upload in chunks but that seemed daunting. Instead, I went back to the developer site and found a set of BASH scripts that use the curl and openssl utilities to handle the heavy lifting.

There are two scripts in the package provided by the developer "nescafe5", hmac and s3. The s3 script is the main script and it calls hmac to calculate the hashes needed to authenticate to S3. Then, it uses curl to upload, download, list, or delete the objects in a bucket. It can also create and destroy buckets.

To use the s3 script, you need to store your Amazon secret key in a text file and set two environment variables. The INSTALL file included with the package has all the details. The only tricky part I ran into, and from the comments on Amazon, other people ran into, is how to create the secret key text file.

If you open a text editor like vim or nano, copy in your secret key and save the file, the editors will add a new line character (hex 0A) to the end of the file. The script requires the file to be exactly 40 bytes, the length of the secret key, or it will complain and stop. To create the text file without a new line, use the echo command:

echo -n "secret-key" > secret-key-file.txt

The -n switch tells echo to not include a new line character and results in a text file of exactly 40 bytes. Once I got the key file created correctly, the s3 script started working, and I was able to upload, download, and list objects in S3.

Here is an example of a test script I used:

#!/bin/bash
# export required variables
export S3_ACCESS_KEY_ID="99XC79990C39996AR999"
export S3_SECRET_ACCESS_KEY="secret-key-file.txt"

# store a file
./s3 put bucketname objectname /path/to/local/file

# list objects in a bucket
./s3 ls bucketname

# download a file
./s3 get bucketname objectname /path/to/local/file

Mac and Linux friendly

I tested the s3 scripts on both OS X Leopard and Red Hat Enterprise Linux 4 and they worked perfectly on both. Amazon Web Services offers powerful solutions to some tough problems. Other services offered include the Elastic Compute Cloud (EC2) for virtual servers, Simple Queue Service (SQS) for passing messages between applications, and Mechanical Turk for delegating work to humans. There is some truly cutting edge stuff going on at Amazon.

Say say say

2008-12-16T10:25:00.001-08:00

The say command tells your Mac to talk back, reciting either a string of text or reading an entire file.

To have the computer read a string of text

Open a Terminal and type:
say "This is a string of text."

To have the computer read text from a file

Open a Terminal and type:
say -f /path/to/text/file

Using a different voice

The speech synthesis system comes with several voices. You can see all the voices available in System Preferences / Speech. The default voice selected in System Preferences is used unless a different one is specified with the -v switch. For example, to use the "Kathy" voice:

say -v Kathy "This is a string of text."

Installing a .dmg application from the command line

2008-12-16T10:24:00.000-08:00

An intrepid reader asked the following question:

How do you install a .dmg package from the command line?

Many applications are distributed as disk images, a compressed binary format. If you double click a disk image in the Finder, it is mounted automatically. Once mounted, installation of the application is typically done by dragging an icon to the Applications folder. The same can be accomplished from the command line using two commands, hdiutil and cp.

The following steps show the installation of a popular VNC client for OS X called "Chicken of the VNC". It can be used as a remote desktop client for Linux, Mac, or Windows hosts.

The download file is named "cotvnc-20b4.dmg". Here are the steps needed to install it remotely from the command line.
note: this technique can be used from a local Terminal window or a remote SSH connection.

Mount the disk image

The first step is to mount (or attach) the disk image. From the command line, use:
hdiutil mount cotvnc-20b4.dmg
I received the following output:

Checksumming Driver Descriptor Map (DDM : 0)…
    Driver Descriptor Map (DDM : 0): verified   CRC32 $767AD93D
Checksumming Apple (Apple_partition_map : 1)…
    Apple (Apple_partition_map : 1): verified   CRC32 $DD66DE0F
Checksumming disk image (Apple_HFS : 2)…
..............................................................................
         disk image (Apple_HFS : 2): verified   CRC32 $EF1F362F
Checksumming  (Apple_Free : 3)…
                   (Apple_Free : 3): verified   CRC32 $00000000
verified   CRC32 $F5A3FFA1
/dev/disk1           Apple_partition_scheme          
/dev/disk1s1         Apple_partition_map             
/dev/disk1s2         Apple_HFS                       /Volumes/Chicken of the VNC

A mounted disk image appears on the Desktop, in the Finder, and more importantly shows up as a directory in /Volumes. In this case, the last line of output from hdiutil showed exactly where the disk image was mounted.

Sometimes when a disk image is mounted, it will prompt you to agree to a license first. In that case, the text that would normally appear in a GUI dialog box instead appears in the Terminal window. Once you scroll to the bottom of the agreement, you can type in Y to continue or N to stop. The Firefox disk image is one example of a package that displays a license before mounting.

Install the application

Use the cp command to copy the application to /Applications:

sudo cp -R "/Volumes/Chicken of the VNC/Chicken of the VNC.app" /Applications

The -R switch means to copy recursively, in other words, copy everything from that location including all subdirectories and files below. It is important to leave off the trailing "/" from the "Chicken of the VNC.app" directory, or the command will not copy the directory itself, just the contents. After entering your password, the application will be installed and ready to use.

Most applications can simply be copied to the /Applications directory. However, some are distributed in a .pkg format and must be installed using the installer command instead of cp. To install a .pkg, use this command:

sudo installer -package /path/to/package -target "/Volumes/Macintosh HD"

Unmount the disk image

To tidy up, return to your home directory and unmount the disk image:

cd ~ hdiutil unmount "/Volumes/Chicken of the VNC/"

You should see this message after the unmount:

"/Volumes/Chicken of the VNC/" unmounted successfully.

Installing applications from a .dmg package at the command line is not something you need to do every day. But it is a nice tool to have if you want to install an application on a remote server or script the installation of a package to a group of desktop Macs.

Updating Apple software from the command line

2008-12-16T10:22:00.000-08:00

Macs check for system updates on their own (by default) and prompt you when there are updates to apply to your system. If you want to update immediately, like after a fresh OS X install, you can check for updates by clicking on the Apple menu icon, then Software Update. However, if you want to update a system remotely -- a frequent need for remote servers -- you can do it from the command line.

The softwareupdate command

Use the softwareupdate (all one word) command to check for updates or install them. Here are some commonly used switches:

Switch	Effect
`--list`	list available updates
`--install`	install all available updates
`--install packagename`	install only packagename
`--schedule on`	turn automatic checking on
`--schedule off`	turn automatic checking off

If you check for new software and there is nothing to apply, you get output like this:

iBook-G4:~ keithw$ softwareupdate --list
Software Update Tool
Copyright 2002-2007 Apple

No new software available.

When updates are available, you get output like this:

iBook-G4:~ keithw$ softwareupdate --list
Software Update found the following new or updated software:
  * iLifeSupport82-8.2
       iLife Support (8.2), 2770K [recommended]
  * QuickTime-7.4.1
       QuickTime (7.4.1), 59940K [recommended] [restart]
  * MacOSXUpd10.5.2-10.5.2
       Mac OS X Update (10.5.2), 349324K [recommended] [restart]

To install updates, use the --install --all option:

iBook-G4:~ root# softwareupdate --install --all
Software Update Tool
Copyright 2002-2007 Apple

Downloading iLife Support       0..20..40..60..80..100
Verifying iLife Support
waiting iLife Support
waiting QuickTime
Downloading Mac OS X Update     0..20..40..60..80..100
Verifying Mac OS X Update
waiting Mac OS X Update
Installing iLife Support        0..20..40..60..80..100
Done iLife Support
Installing QuickTime    0..20..40..60..80..100
Done QuickTime
Installing Mac OS X Update      0..20..40..60..80..100
Done Mac OS X Update
Done.

You have installed one or more updates that requires that you restart your
computer.  Please restart immediately.

Syncing Palm devices with the Mac via Bluetooth

2008-12-16T10:21:00.000-08:00

Until I figured out how to sync my Palm based Treo smartphone with the Palm Desktop on my iBook, I had no use for bluetooth. Before then, bluetooth was one of those technologies that was around but that I largely ignored. Now, the thought of syncing via USB cable seems so 20th century. As a bonus, I cover how to transfer camera images from the Treo to the Mac via bluetooth.

This article documents all the steps needed to set up wireless syncing via bluetooth between a Treo and Palm Desktop for the Mac. Since there are a variety of Palm based devices and OS X versions, it is offered as is. Just because it "works for me" doesn't mean it will work for you. At the least, you may get some useful hints for your situation.

Hardware

My main Mac is currently an iBook G4 1.33 Ghz running Leopard (OS X 10.5.1). My Palm device is a Treo 650 smartphone running Palm OS Garnet (5.4.8). Both devices have native bluetooth wireless hardware.

Bluetooth is a low power, short range radio technology (wireless) designed to link devices together in a personal area network. Bluetooth 1.x operates over distances of about 10 meters with a data transfer rate of 720 kbps. While the low data rate is useless for large files, it works great for syncing Palm data or sending a few images back and forth.

Installing Palm Desktop

My first stop was the Palm web site where I downloaded the latest Palm Desktop software for Mac. The version I downloaded was 4.2.1 rev D which came in disk image (.dmg) format. I mounted the disk image and ran the installer. At the end of the installation, a reboot is required, probably to load the background processes cleanly.

Bluetooth configuration on the Mac

After the reboot, I visited System Settings and went into the Sharing settings. There I enabled bluetooth sharing and required pairing for all actions. Pairing is the process of introducing bluetooth devices to one another using a shared passkey. I learned later that pairing was not really required for Palm syncing, but I prefer to use it for file transfer so I checked all the pairing boxes.

Configuring HotSync manager

Staying on the Mac, go to Applications and open the Palm folder containing all the Palm Desktop applications. Run the HotSync Manager, go to the Connection Settings tab, and enable the bluetooth-pda-sync-port.

The dialog box on my screen cut off the name, but if you select it, you can see the full name beneath the list of selections. It is the second choice in the list, below the bluetooth-modem.

As a final check of the Mac set up, I opened a Terminal and ran the following command to verify that the Palm Transport Monitor was running in the background:

iBook-G4:~ keithw$ ps aux | grep Palm keithw 130 0.1 0.3 154888 3468 ?? S 6:01PM 0:04.04 /System/Library/Frameworks/Carbon.framework/Versions/A/Support/LaunchCFMApp /Applications/Palm/Palm Desktop/Contents/Resources/Palm Desktop Background.app/Contents/MacOSClassic/Palm Desktop Background keithw 131 0.0 0.3 118108 3628 ?? S 6:01PM 0:00.79 /System/Library/Frameworks/Carbon.framework/Versions/A/Support/LaunchCFMApp /Applications/Palm/Transport Monitor/Contents/MacOSClassic/Transport Monitor

Bluetooth configuration on the Palm Treo

On the Treo, go to the Communications group and open the Bluetooth application (it uses the Bluetooth icon). Make sure bluetooth is "On" and Discoverable is set to "Yes".

Then, click the "Setup Devices" button and on the next screen, click the "HotSync Setup" button. This starts a wizard-like procedure with 4 steps. The instructions use PC terminology and may appear confusing, but all of the Mac side set up should be done. Click Next to continue.

Step 1: says to create a virtual serial port on your PC, just click Next.

Step 2: We've already enabled the bluetooth-pda-sync-port in HotSync Manager, so just click Next.

Step 3: The local serial port is already enabled on the Mac, so click Next.

Step 4: click on the Launch HotSync button, it should start the first HotSync immediately. The "HotSync Progress" dialog should appear on the Mac indicating the first HotSync is under way.

Depending on how much data you have on your Palm, the first HotSync may take a long time. It has to transfer all of your data over bluetooth to the Mac. After the first HotSync is complete, only changed data will be transfered and HotSyncs will typically be quick.

Transferring files via bluetooth

Like most cell phones today, the Treo has a built in camera. Even though it is low resolution, it is still useful and convenient. I was looking for a zero cost way to transfer photos from the Treo to the Mac. Bluetooth was the answer.

Bluetooth configuration on the Mac

See the section above with the same title for details and a screenshot for configuring bluetooth.

While pairing may not be absolutely required, I think it is the best practice to require it. Otherwise, anyone wandering by with a bluetooth device could upload files to your Mac. I can think of a lot of bad scenarios that might arise out of that configuration.

Pairing the Treo and Mac

On the Treo, open the Bluetooth application (it uses the Bluetooth icon). Make sure bluetooth is "On" and Discoverable is set to "Yes". Then, click the "Trusted Devices" button, then "Add Device".

Select the Mac (it must also be discoverable) and click OK.

The Treo will prompt for a passkey. Enter any sequence of digits (I only used 4 digits). Enter the same passkey on the Mac when prompted. Pairing is a one time set up procedure and you won't need to remember the passkey after they are paired.

Sending an image (or any file) to the Mac

To send a camera image file, open any image on the Treo in the Media application. Then, open the Media menu and select Send...

In the Send with dialog, choose Bluetooth..., then select the Mac you are paired with, then OK. The file will be sent and the Mac will prompt you to accept it.

After you accept it, the file is saved in the folder you configured in bluetooth sharing, the "Download" folder in my case. It will have a file name similar to "Photo_112307_003.jpg".

How to enable the OS X root user

2008-12-14T18:01:00.000-08:00

The following information was found at spy-hill.com.

This is the easiest method to enable the "root" account on a Mac if you are more of a CLI person:

Log in on the Admin account. (Your normal, every day user account should not have administrative privileges!).
Open up a command shell in the Terminal application with
Macintosh HD -> Applications -> Utilities -> Terminal.

At the command prompt type this command:
% sudo passwd root Enter Password: Changing password for root New password: Verify password:
The first password you are asked for is the already existing password for the Admin account, to prove that you are authorized to make changes to this system. After that, you enter the new root password (twice, for verification). That is all, the "root" account is now enabled, with that password.

Screenshots from the command line

2008-12-14T15:54:00.000-08:00

Use the screencapture utility to initiate a screenshot from the command line.

Take a screenshot and save the result to a file

This command can be used to take a screenshot with a 10 second delay (-T 10) and save the result in jpg format (-t jpg) to a file named screen.jpg:

screencapture -T 10 -t jpg screen.jpg

If the file format is not specified, a png file is saved. The time delay lets you change desktops if you are using Spaces (the multiple desktop feature in Leopard) before the screenshot is taken.

How to find out which files were installed with a Mac package

2008-12-14T10:08:00.001-08:00

Installing a Mac package is usually a process of dragging an icon from an installer to the Applications folder. This creates a new directory in the Applications folder with the application files inside it. Sometimes, additional files are installed in other places. To see everything that was installed, use the /Library/Receipts directory and the lsbom utility.

An applications that follows Mac standards will leave a record of what was installed in the /Library/Receipts/app-name directory.

Within that directory, there should be a Contents subdirectory, and inside that, an Archive.bom file. The .bom file is a "bill of materials" with a list of all the files that belong to that application, the full path to their location, and other details like the UNIX permissions, file size, and checksum.

However, the bill of materials is a binary file. To extract information from it, use the list bill of materials command line utility, lsbom.

To list all files and related information in a BOM file:

lsbom BOMfile

To list only the files without related information in a BOM file:

lsbom -s BOMfile

Using hdiutil

2008-12-14T10:06:00.000-08:00

The hdiutil program is a native Apple command line utility for working with disk images. It uses the DiskImages framework. Disk images (usually with a .dmg, .img, or .iso file name extension) are often used for distributing programs and for burning CDs/DVDs.

Creating a disk image from a folder

hdiutil create test.dmg -srcfolder /path/to/folder/

Once the disk image is created, it can be mounted, copied, or sent like any other file.

Mounting a disk image

To mount (or attach) a disk image, use:

hdiutil mount test.dmg

A mounted disk image appears on the Desktop, in the Finder, and shows up as a directory in /Volumes. All removable media, such as CDs, DVDs, external disks, and USB flash drives are mounted in /Volumes.

Unmounting a disk image

To unmount (or detach) a disk image, use:

hdiutil unmount /dev/device-name

The device name is usually something like /dev/disk3s2. You can also unmount it using the /Volumes/mountpoint if you know where it was mounted:

hdiutil unmount /Volumes/mountpoint

Burning an ISO to CD (or DVD)

First, load a blank CD, then:

hdiutil burn cd-image.iso

Create an encrypted disk image

This creates a 10 MB encrypted disk image and internally formats it as a journaled HFS+ file system (the OS X default):

hdiutil create -encryption -size 10m -volname encdata test.dmg -fs HFS+J

During the creation of the disk image, you will be prompted for a password that will allow access to the contents of the disk image. You must remember the password or anything you put into the image will be lost.

Apache Authentication and Authorization using LDAP

2008-12-14T07:47:00.000-08:00

The Lightweight Directory Access Protocol (LDAP) is frequently used to implement a centralized directory server. There are two popular open source LDAP solutions, OpenLDAP and Red Hat Directory Server. According to the Apache documentation, Novell LDAP and iPlanet Directory Server are also supported. This article focuses on OpenLDAP, but the concepts and examples should be applicable to the others.

note: originally published October 31, 2007 on linux.com

Information in OpenLDAP

LDAP was designed as a simplified, or "lightweight", version of the ITU-T X.500 directory specification. The default set of schemas contain all of the information you would find in traditional Linux system files like /etc/passwd and /etc/group, or Sun's Network Information System (NIS). The schemas are malleable and are often extended to contain additional demographic information or customized for specific applications.

Following is an example of a typical LDAP user record in LDAP Data Interchange Format (LDIF):

dn: uid=keithw,ou=People,dc=company,dc=com
uid: keithw
cn: Keith Winston
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
userPassword: {crypt}$1$M/PZEwdp$KHjSay8JILX01YAHxjfc91
shadowLastChange: 13402
shadowMax: 99999
shadowWarning: 7
loginShell: /bin/bash
uidNumber: 2741
gidNumber: 420
homeDirectory: /home/keithw
gecos: Keith Winston

The LDAP database can be queried with a number of tools, including the command line ldapsearch program, one of the standard OpenLDAP utilities. If you are new to LDAP, the terminology and syntax may be difficult at first. Taking the time to learn the LDAP search syntax will pay off later if you want to craft advanced policies using non-standard attributes.

Configuring Apache 2.2

Apache modules have been available for LDAP since at least version 1.3. If you have used mod_auth_ldap in the past, you should be aware that the bundled authentication and authorization modules have been refactored in version 2.2. The latest LDAP modules are loaded with these directives, usually in the httpd.conf file:

LoadModule ldap_module /path/to/mod_ldap.so LoadModule authnz_ldap_module /path/to/mod_authnz_ldap.so

Once the modules are loaded, you can control access by querying the directory for particular attributes. The key directive to point Apache at the LDAP server is AuthLDAPUrl. A generic AuthLDAPUrl directive looks like this:

AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid

It defines the LDAP server, the base distinguished name (DN), the attribute to use in the search (usually Uid within the People organizational unit). For complex policies, you may need extra search filters.

The next few sections show working examples of directives to enforce common policies. Each set of directives can be placed in the main Apache configuration file or in .htaccess files.

Any valid user

This set of directives allows access to the current directory to all valid users in the LDAP directory. Apache will ask the browser for a user ID and password and check that against the directory. If you are familiar with Apache Basic Authentication, there are only a few new directives to learn.

Order deny,allow
Deny from All
AuthName "Company.com Intranet"
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid
Require valid-user
Satisfy any

AuthBasicProvider ldap is necessary so Apache knows to query an LDAP directory instead of a local file.

AuthzLDAPAuthoritative off must be explicitly set because the default setting is "on" and authentication attempts for valid-user will fail otherwise. This is a tricky setting because other policies, like Require ldap-user, need the setting to be "on". Setting this value off also allows other authentication methods to mixed with LDAP.

The Satisfy any directive is not strictly required in this case because we are only testing one condition.

List of users

This set of directives allows access to the current directory to the users listed in the Require ldap-user directive.

Order deny,allow
Deny from All
AuthName "Company.com Intranet"
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid
Require ldap-user keithw joeuser
Satisfy any

AuthzLDAPAuthoritative on could be omitted since the default setting is "on", but left here for clarity.

Note the AuthLDAPUrl setting does not change. As in previous examples, it searches the directory for a matching Uid.

Member of a group

This set of directives allows access to the current directory to users who are either primary or secondary members of the group specified in the Require ldap-group directive.

The group configuration may be the most difficult due to the schema design of directories that were converted from NIS (as mine was). Referring back to the user LDIF record, notice the gidNumber attribute has a value of 420, the number assigned to the infosys group in my directory. It corresponds to the primary group of the user. However, the LDAP entry for each group only lists users who are secondary members of the group, using the memberUid attribute. See below for a snippet of a group record:

dn: cn=infosys,ou=Group,dc=company,dc=com
objectClass: posixGroup
gidNumber: 420
memberUid: user1
memberUid: user2
memberUid: user3
...

We need another test, Require ldap-attribute, to pick up the primary users of the group because they are not listed with the group itself. Here are the Apache directives:

Order deny,allow
Deny from All
AuthName "Company.com Intranet"
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid
AuthLDAPGroupAttribute memberUid
AuthLDAPGroupAttributeIsDN off
Require ldap-group cn=infosys,ou=Group,dc=company,dc=com
Require ldap-attribute gidNumber=420
Satisfy any

AuthzLDAPAuthoritative on could be omitted since the default setting is "on", but left here for clarity.

AuthLDAPGroupAttribute memberUid indicates which attibute in the LDAP group record to match with the Uid. In this case, memberUid. A group record contains one memberUid attribute for each (non primary) member of the group.

AuthLDAPGroupAttributeIsDN off tells Apache to use the distinguished name of the client when checking for group membership. Otherwise, the username will be used. In my OpenLDAP directory, only the username was from NIS. The default setting is "on" so setting it off was required. An LDAP directory may store the entire distinguished name, so this setting may be change based on your directory.

Require ldap-group grants access to members of the "infosys" group. For multiple groups, add an additional directive for each.

Require ldap-attribute gidNumber=420 handles the primary users of group 420, the "infosys" group. Without this condition, primary users would be denied access. For multiple groups, add an additional directive for each.

The Satisfy any directive is required because we are testing multiple conditions and want the successful test of any condition to grant access.

Combination of users and groups

This example is a union of the user and group directives, but otherwise, there is nothing new.

Order deny,allow
Deny from All
AuthName "Company.com Intranet"
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid
AuthLDAPGroupAttribute memberUid
AuthLDAPGroupAttributeIsDN off
Require ldap-group cn=infosys,ou=Group,dc=company,dc=com
Require ldap-attribute gidNumber=420
Require ldap-user keithw joeuser
Satisfy any

Debug and deploy

Testing LDAP authentication from a web browser can be frustrating because the only thing you know is that access was granted or not. You don't get any kind of feedback on why something did not work. For verbose information on each step in the process, set the "LogLevel debug" option in Apache.

With debugging active, Apache will record the connection status to the LDAP server, what attributes and values were requested, what was returned, and why conditions were met or not met. It can be invaluable in fine tuning LDAP access controls.

Using diskutil

2008-12-13T19:50:00.000-08:00

The diskutil program is a native Apple command line utility for manipulating disks, partitions, and RAID sets. This includes magnetic hard disks, CDs/DVDs, and flash drives. Most options, except "list", require root access.

Finding out about disks in your system

For information on all available disks and their partitioning, use:
diskutil list

For more detailed information on a particular disk or partition, use:
diskutil info disk-or-partition

The default Apple partitioning scheme uses the last physical partition on a disk for storing data. Here is sample output from diskutil list showing a hard disk and a CD. The UNIX device name is shown first, along with the contents of each partition:

/dev/disk0
  #:                       TYPE NAME                    SIZE       IDENTIFIER
  0:     Apple_partition_scheme                        *74.5 Gi    disk0
  1:        Apple_partition_map                         31.5 Ki    disk0s1
  2:                  Apple_HFS Macintosh HD            74.4 Gi    disk0s3
/dev/disk1
  #:                       TYPE NAME                    SIZE       IDENTIFIER
  0:        CD_partition_scheme                        *718.1 Mi   disk1
  1:     Apple_partition_scheme                         625.3 Mi   disk1s1
  2:        Apple_partition_map                         31.5 Ki    disk1s1s1
  3:                  Apple_HFS Dungeon Siege Disc 2    625.0 Mi   disk1s1s2

Here is sample output from diskutil info on a disk partition:

diskutil info /dev/disk0s3
  Device Identifier:        disk0s3
  Device Node:              /dev/disk0s3
  Part Of Whole:            disk0
  Device / Media Name:      Untitled 3

  Volume Name:              Macintosh HD
  Mount Point:              /
  File System:              Journaled HFS+
                            Journal size 8192 KB at offset 0x256000
  Owners:                   Enabled

  Partition Type:           Apple_HFS
  Bootable:                 Is bootable
  Media Type:               Generic
  Protocol:                 ATA
  SMART Status:             Verified
  Volume UUID:              EE8C4FFD-6C4B-302D-B096-DCF81D4E13FB

  Total Size:               74.4 Gi (79892103168 B) (156039264 512-byte blocks)
  Free Space:               26.8 Gi (28731383808 B) (56115984 512-byte blocks)

  Read Only:                No
  Ejectable:                No
  Whole:                    No
  Internal:                 Yes

Checking partitions for integrity and fixing them

You can use diskutil to check the file system data structure of a partition (e.g., /dev/disk0s3) with:
diskutil verifyVolume partition

If errors are you found, you can fix them with:
diskutil repairVolume partition

Checking partitions for UNIX permission problems and repairing them

You can use diskutil to check the UNIX permissions on a partition with:
diskutil verifyPermissions partition

If errors are you found, you can fix them with:
diskutil repairPermissions partition

If permissions get accidentally changed on some system files, it could cause strange behavior or disable certain features of the system.

Finding out about RAID sets

RAID is usually used in servers to provide additional protection from hard disk failure. For information on RAID sets, use:
diskutil listRAID

Other diskutil options

In addition to the options listed above, diskutil can be used to reformat disks or partitions, erase writable CDs/DVDs, securely erase data, etc. Here are some of the other features:

u[n]mount - Unmount a single volume
unmountDisk - Unmount an entire disk (all volumes)
eject - Eject a removable disk
mount - Mount a single volume
mountDisk - Mount an entire disk (all mountable volumes)
eraseDisk - Erase an existing disk, removing all volumes
eraseVolume - Erase an existing volume
reformat - Reformat an existing volume
eraseOptical - Erase an optical media (CD/RW, DVD/RW, etc.)
zeroDisk - Erase a disk, writing zeros to the media
randomDisk - Erase a disk, writing random data to the media
secureErase - Securely erase a disk or freespace on a volume
resizeVolume - Resize a volume, increasing or decreasing its size

See the man page for even more!

Copy and paste inside a shell script

2008-12-13T19:26:00.000-08:00

The Terminal application supports copy and paste from the Edit menu or using the standard keyboard shortcuts. In addition, two native Apple command line programs, pbcopy and pbpaste, allow you to copy and paste text from the clipboard within a shell script.

pbcopy takes standard in and places it in the specified pasteboard (the clipboard). For example, to capture the output of the ls -1 command, pipe the output to pbcopy:

ls -1 | pbcopy

Now, the results can be pasted into any program that supports the clipboard.

pbpaste writes the contents of the pasteboard (the clipboard) to standard out. For example, to paste the contents of the clipboard into a text file called "output.txt", use:

pbpaste > output.txt

Defaults -- setting preferences from the command line

2008-12-13T19:23:00.000-08:00

The defaults program allows users to read, write, and delete Mac OS X user preferences from the command line. OS X applications use the defaults system to record user preferences and other information that must be maintained when the applications aren't running. Setting preferences is usually done (more easily) from the GUI, but some options are not available in the GUI.

Property lists

User preferences and application settings are stored in files called property lists (with an extension of .plist). These files can store binary values, so they can sometimes look meaningless if you view the raw file. Aside from the Preferences dialogs, property list files can be updated with the Property List Editor application (part of the developer tools), or with the defaults command line program.

User defaults belong to domains, which typically correspond to individual applications. For example, the domain for Firefox is org.mozilla.firefox with the settings saved in a file called org.mozilla.firefox.plist. Each domain has a dictionary of keys and values representing its defaults; for example, "Default Font" = "Helvetica". Keys are always strings, but values can be complex data structures comprising arrays, dictionaries, strings, and binary data.

User plist files are saved in ~/Library/Preferences/, while system wide plist files are stored in /Library/Preferences/.

When you update a setting using defaults, it only affects the current user.

View preferences

To see all user preferences, use:
defaults read

The output list will be very long because every setting for every application will be printed.

To see all Firefox user preferences, use:
defaults read org.mozilla.firefox

Note that Firefox stores most of it's settings in Firefox specific files, and not in Apple's defaults system. The same is true in Windows where most of the Firefox settings are not in the Windows registry.

Update or delete a preference

To update a key use:
defaults write domain { key 'value' }

Of course, you have to know the name of the key and the acceptable values you can use ahead of time.

To delete a key use:
defaults delete domain key

Nvram

2008-12-13T17:57:00.000-08:00

The nvram utility is used to view and set Open Firmware NVRAM variables. Open Firmware, (aka non-volatile RAM), controls system boot options. To change a variable, you must run nvram as root.

Viewing Open Firmware variables

There are about 50 Open Firmware variables. To view all variables, use:
nvram -p

Setting Open Firmware variables

You must be root to set a variable. To set a variable, assign a value with an equals sign:
nvram scroll-lock=false

After setting a variable, they are saved during the next clean restart or shutdown.

To set multiple variables at once, list them in a text file in variable=value format, one per line.
Then, feed them all to nvram with:
nvram -f text-file

You can also create new variables by simply assigning a value to a new variable name. It won't have any meaning to the system unless some application reads it, but it is a nice option.

Deleting Open Fireware variables

To delete a variable, use:
nvram -d variable

Sysctl

2008-12-13T14:38:00.000-08:00

Sysctl is a kernel utility that allows you to query and set kernel variables. It is rare that you need to tinker with a running OS X kernel, but special applications may require it.

Viewing kernel variables

There are over 100 kernel state variables. To view a specific variable, use:
sysctl variable-name

To view all variables, use:
sysctl -a

Setting kernel variables

You must be root to set a kernel variable. To set a kernel variable, use the -w switch:
sysctl -w variable-name=value

If you are running OS X as a file server and have a lot of people connecting and sharing files, you might want to increase the maximum number of open files allowed by the kernel. This can be tuned using the kern.maxfiles variable. The default value is 12288. To increase it to 50000, use:
sysctl -w kern.maxfiles=50000

Diagnostic command line tools

2008-12-13T14:21:00.001-08:00

OS X comes with a number of command line tools to measure system performance and individual application performance. Using these tools, you can track down bottlenecks and discover misbehaving applications -- those acting like rude, thoughtless, little pigs.

top

The top program shows a full screen of stats on overall CPU usage, memory usage, load average, and how many resources each process is using. The default mode for top is an interactive display that updates once per second. Applications are sorted by CPU usage in descending order. To quit while in interactive mode, type q.

Processes:  64 total, 3 running, 1 stuck, 60 sleeping... 204 threads   16:21:14
Load Avg:  0.52, 0.43, 0.25     CPU usage:  9.3% user, 22.9% sys, 67.8% idle
SharedLibs: num =   27, resident = 11.0M code, 1.02M data, 3.44M LinkEdit
MemRegions: num = 11872, resident =  388M + 9.48M private,  111M shared
PhysMem:   117M wired,  393M active,  374M inactive,  885M used,  138M free
VM: 5.57G + 21.5M   284085(0) pageins, 19704(0) pageouts

 PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
23002 mdimport     0.1%  0:00.38   4    62    52   996K  4.27M  3.01M  40.0M
22998 top         12.3%  0:24.47   1    19    22   476K   476K   900K  27.0M
22977 lookupd      0.0%  0:00.09   2    34    37   400K  1.06M  1.16M  28.5M
22971 mdimport     0.0%  0:00.31   3    60    46   864K  3.37M  2.76M  39.0M
19117 firefox-bi  10.1% 12:34:42  10   196   781   143M  59.6M   155M   983M
18711 bash         0.0%  0:00.14   1    14    17   216K   884K   748K  27.2M
18710 login        0.0%  0:00.02   1    16    37   124K   516K   460K  26.9M
17409 httpd        0.0%  0:00.00   1    11    87   104K  2.47M   476K  31.9M
7211 usbmuxd      0.0%  0:00.02   2    23    23   164K   436K   284K  27.0M
7193 iTunesHelp   0.0%  0:01.38   2    55    99   608K  4.67M  1.55M   108M
4618 Terminal     4.2% 27:15.18  10   173   228  3.91M+ 16.5M+ 9.18M+  135M+
 717 AppleSpell   0.0%  0:00.28   1    32    36   284K  2.06M   860K  37.7M
 712 System Eve   0.0%  0:03.10   1    62   109   928K  7.02M  1.30M   109M
 682 cupsd        0.0%  0:58.79   2    30    25   568K  1.03M   844K  27.8M
 380 DashboardC   0.0%  0:56.12   3    78   160  3.59M  6.57M  4.13M   116M
 379 DashboardC   0.0%  0:07.55   4    86   158  5.75M  6.66M  6.56M   115M

Top can also be run in batch mode where it writes output to the terminal a set number of times, then quits. This is ideal for feeding data to a parsing program for generating your own stats or filtering. For example, to show top output once, use:

top -l 1
note: top minus el one

To sort the output by CPU usage, use:

top -l 1 -o cpu

To sort the output by resident memory usage (physical memory), use:

top -l 1 -o rsize

To sort the output by virtual memory usage (total memory), use:

top -l 1 -o vsize

fs_usage

Report system calls and page faults related to filesystem activity in real-time. It is a continuous display and if unfiltered, is overwhelming. It is usually best to limit the output to a single process or to watch network activity. It must be run as root.

To limit the output to a process or command name, include it at the end of the command:

fs_usage 19117

To limit the output to just network related system calls, use:

fs_usage -f network

To limit the output to just filesystem related system calls, use:

fs_usage -f filesys

vm_stat

The vm_stat program displays Mach (the kernel) virtual memory statistics. It is useful to seeing if your system is memory starved.

Here is the output from vm_stat:

Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                    63416.
Pages active:                 103005.
Pages inactive:                68102.
Pages wired down:              27621.
"Translation faults":      308932364.
Pages copy-on-write:         7961115.
Pages zero filled:         204818181.
Pages reactivated:           1571266.
Pageins:                      279532.
Pageouts:                      19704.
Object cache: 1693448 hits of 4581511 lookups (36% hit rate)

If Pages free is low, you need more memory to run all your applications.

You can add an interval (in seconds) as a parameter to vm_stat and it will show your memory and paging activity in real time. For example, vm_stat 1 will output a new line every second -- useful to see if memory is taking a beating with a lot Pageins/Pageouts, but it won't pinpoint an offending application.

30 days with JFS

2008-12-13T12:17:00.000-08:00

The Journaled File System (JFS) is a little known file system open sourced by IBM in 1999 and available in the official kernel sources since 2002. It originated inside IBM as the standard file system on the AIX line of UNIX servers, and later OS/2. IBM ships JFS2 with AIX today. Despite its pedigree, JFS has not received the publicity or widespread usage of other Linux file systems like EXT2/3 and ReiserFS. To learn more about JFS, I installed it as my root file system.

note: originally published September 14, 2007 on linux.com

Features

I installed Slackware 12 on a laptop choosing JFS as the file system during installation. I performed no special partitioning and created one JFS file system to hold everything. Installation was uneventful and the system booted normally from Grub. Not all distributions offer JFS as an install option, and some may not have JFS compiled into their default kernels. While Fedora and SUSE can use JFS, they both default to EXT3. Slackware, Debian, Ubuntu, and their derivatives are good choices if you want to try JFS.

One of the first things I noticed was the absence of a lost+found directory, a relic of lesser file systems.

JFS is a fully 64-bit file system. With the default block size of 4 KB, it supports a maximum file system size of 4 petabytes (less if smaller block sizes are used). The minimum file system size supported is 16 MB. The JFS transaction log has a default size of 0.4% of the aggregate size, rounded up to a megabyte boundary. The maximum size of the log is 32 MB. One interesting aspect of the layout on disk is the fsck working space, a small area allocated within the file system for keeping track of block allocation if there is not enough RAM to track a large file system at boot time.

JFS dynamically allocates space for disk inodes, freeing the space when it is no longer required. This eliminates the possibility of running out of inodes due to a large number of small files. As far as I can tell, JFS is the only file system in the kernel with this feature. For performance and efficiency, the contents of small directories are stored within the directory's inode. Up to eight entries are stored in-line within the inode, excluding the self [.] and parent [..] entries. Larger directories use a B+tree keyed on name for faster retrieval. Internally, extents are used to allocate blocks to files, leading to compact, efficient use of space even as files grow in size. This is also available in XFS, and is a major new feature in EXT4.

JFS supports both sparse and dense files. Sparse files allow data to be written to random locations within a file without writing intervening file blocks. JFS reports the file size as the largest used block, while only allocating actually used blocks. Sparse files are useful for applications that require a large logical space but only use a portion of the space. With dense files, blocks are allocated to fill the entire file size, whether data is written to them or not.

In addition to the standard permissions, JFS supports basic extended attributes, such as the immutable (i) and append-only (a) attributes. I was able to successfully set and test them with the lsattr and chattr programs. I could not find definitive information on access control list support under Linux.

Logging

The main design goal of JFS was to provide fast crash recovery for large file systems, avoiding the long file system check (fsck) times of older UNIX file systems. That was also the primary goal of file systems like EXT3 and ReiserFS. Unlike EXT3, journaling was not an add-on to JFS, but baked into the design from the start. For high performance applications, the JFS transaction log file can be created on an external volume if specified when the file system is first created.

JFS only logs operations on meta-data, maintaining the consistency of the file system structure, but not necessarily the data. A crash might result in stale data, but the files should remain consistent and usable.

Here is a list of the file system operations logged by JFS:

File creation (create)
Linking (link)
Making directory (mkdir)
Making node (mknod)
Removing file (unlink)
Rename (rename)
Removing directory (rmdir)
Symbolic link (symlink)
Truncating regular file

Utilities

A suite of utilities is provided to manage JFS file systems. You must be the root user to use them.

Utility	Description
jfs_debugfs	shell based JFS file system editor, allows changes to the ACL, uid/gid. mode, time, etc. You can also alter data on disk, but only by entering hex strings, not the most efficient way to edit a file.
jfs_fsck	replay the JFS transaction log, check and repair a JFS device, should on ly be run on an unmounted or read only file system, run automatically at boot.
jfs_fscklog	extract a JFS fsck service log into a file. `jfs_fscklog -e /dev/hd a6` extracts the binary log to file fscklog.new, to view, use `jfs_fscklog -d fscklog.new`
jfs_logdump	dump the journal log, a plaint text file that shows data on each transac tion in the log file.
jfs_mkfs	create a JFS formatted partition, use the `-j journal_device` option to create an external journal (1.0.18 or later)
jfs_tune	adjust tunable file system parameters on JFS. I didn't find options that looked like they might improve performance, the `-l` option lists th e superblock info

Here is what a dump of the superblock information looks like:

root@slackt41:~# jfs_tune -l /dev/hda6
jfs_tune version 1.1.11, 05-Jun-2006

JFS filesystem superblock:

JFS magic number:       'JFS1'
JFS version:            1
JFS state:              mounted
JFS flags:              JFS_LINUX  JFS_COMMIT  JFS_GROUPCOMMIT  JFS_INLINELOG
Aggregate block size:   4096 bytes
Aggregate size:         12239720 blocks
Physical block size:    512 bytes
Allocation group size:  16384 aggregate blocks
Log device number:      0x306
Filesystem creation:    Wed Jul 11 01:52:42 2007
Volume label:           ''

Crash Testing

White papers and man pages are no substitute for the harsh reality of a server room. To test the recovery capabilities of JFS, I started crashing my system (forced power off) with increasing workloads. I repeated each crash twice to see if my results were consistent.

Crash workload	Recovery
console (no X) running text editor with one open file	about 2 seconds to replay the journal log, changes I had not saved in th e editor were missing but the file was intact
X window system with KDE, gimp, nvu, and text editor in xte rm all with open files	about 2 seconds to replay the journal log, all open files were intact, u nsaved changes were missing
X window system with KDE, gimp, nvu, and text editor all with open files , plus a shell script that inserted records into a MySQL (ISAM) table. The scri pt I wrote was an infinite loop and I let it run for a couple of minutes to make sure some records were flushed to disk.	about 3 seconds to replay the journal log, all open files intact, databa se intact with a few thousand records inserted, but the timestamp on the table f ile had been rolled back one minute.

In all cases, these boot messages appeared...

**Phase 0 - Replay Journal Log
-|----- (spinner appeared for a couple of seconds, then went away)
Filesystem is clean

Throughout the crash testing, no file system corruption occurred and the longest log replay time I experienced was about 3 seconds.

Conclusion

While my improvised crash tests were not a good simulation a busy server, JFS did hold up well, and recovery time was fast. All file level applications I tested, like tar and rsync, worked flawlessly and lower level programs like truecrypt also worked as expected. After 30 days of kicking and prodding, this gave me a high level of confidence in JFS and I am content trusting my data to it. JFS may not have been marketed as effectively as others, but is a solid choice in the long list of quality Linux file systems.

MacPorts, Up and Running

2008-12-13T12:13:00.000-08:00

MacPorts is a system designed to let you easily install and manage free, open source software on OS X. Until late in 2006, the project was named DarwinPorts. There are disk image downloads available for 10.3 (Panther), 10.4 (Tiger), and 10.5 (Leopard). If you have a different version, you can download the source code and compile it.

updated for MacPorts 1.6 and Leopard on January 16, 2008

The Ports concept

If you are new the "ports" concept, the way the system works may be hard to grasp. The ports system was developed in the BSD community as a way to easily install and manage software on any BSD system. It was designed to automate the manual process programmers followed to port software from one Unix-like system to another. Most of the software in the ports repository is written in the C language and requires a C compiler.

When a particular package is requested, the ports system checks the version of the system it is running on, the version of the available source code, then downloads the source, applies any patches needed for it run on the local system, compiles it, and installs it. The design of the ports system only requires package maintainers to write scripts that handle any differences in the local system (like library versions, hardware architecture, compiler versions, etc.). The heavy lifting is done by the ports system itself.

This system is ingenious and works well in most situations, but you may occasionally run into a package that doesn't work. If that happens, it is usually because the source program was recently updated and the port scripts haven't caught up.

Universal ports

An Apple enhancement to the original ports system allows some software to be built (compiled) as universal binaries. This option may be desirable if you share binaries between Intel and PPC Macs, or plan to upgrade soon and don't want to install your ports again.

Installing MacPorts

According to the MacPorts wiki, there are three critical steps that must be done prior to installing MacPorts:

Install Xcode Tools
Install XWindows (X11)
Set the shell environment

Both Xcode Tools (which includes the compiler), and X11 are available on the Mac DVDs. They can also be downloaded from the Apple developer web site. I had already installed Xcode and XWindows, so the only thing I needed to do was modify my $HOME/.bash_login file to include the /opt/local directories in my PATH and set the DISPLAY variable with "export DISPLAY=:0.0". See the wiki page for additional details.

After setting my shell environment, I downloaded version 1.6 of the MacPorts disk image for 10.5 (Leopard) from the download area. I mounted the disk image and ran the .pkg installer to install MacPorts. It completed with no errors.

If you want to tweak the port system configuration, the default location of the global config file is /opt/local/etc/macports/macports.conf.

Getting started

Once the ports system was installed, I followed the wiki recommendation and updated MacPorts itself with this Terminal command:

sudo port -d selfupdate

Next, I installed a simple program to see what kind of output the system produced. Here is the output from installing dos2unix, a program that changes the line endings of a text file from DOS to UNIX format.

sudo port install dos2unix
Password:
--->  Fetching dos2unix
--->  Attempting to fetch dos2unix-3.1.tar.gz from http://fresh.t-systems-sfr.com/linux/src/
--->  Verifying checksum(s) for dos2unix
--->  Extracting dos2unix
--->  Configuring dos2unix
--->  Building dos2unix with target all
--->  Staging dos2unix into destroot
--->  Installing dos2unix 3.1_0
--->  Activating dos2unix 3.1_0
--->  Cleaning dos2unix

When the port command finished, I had a freshly compiled dos2unix program sitting in /opt/local/bin/. Everything is nicely isolated in /opt/local/ to prevent any port program from interfering with the original Apple system.

One of more powerful features of ports is that it understands program dependencies. If you ask it to install a program that requires other programs or libraries to run, the ports system first installs the required components, then completes your request. For example, I wanted to install wget, a useful command line tool to download files or mirror web sites. I started the install with:

sudo port install wget

After starting it, I walked away. When I came back 30 minutes later, ports was still still installing dependencies. There were five dependencies that got installed first:

expat
libiconv
gettext
zlib
openssl

If you start a port install and your system seems sluggish, check to see if ports is still downloading and compiling dependencies.

Uninstalling MacPorts

Because of the way MacPorts files are isolated, it can be easily removed. To uninstall both the MacPorts system and all ports installed, the wiki lists the following files and directories to delete:

/opt/local/
/Applications/MacPorts/
/Library/Tcl/macports1.0/
/Library/LaunchDaemons/org.macports.*
/Library/StartupItems/DarwinPortsStartup

To my surprise, my installation did not include these files and directories:

/Applications/MacPorts/
/Library/LaunchDaemons/org.macports.*
/Library/StartupItems/DarwinPortsStartup

Since the system appears to be fully functional, it is possible that this part of the wiki is slightly out of date.

Update January 10, 2008: Seth Milliken wrote in and noted that these directories are only created if you install a port that uses them, e.g. installing a Cocoa app will create /Applications/MacPorts/.

Common port commands

Here are some commands you may find useful while learning to use the ports system.

Command	Result
port install package	installs a package
port uninstall package	uninstalls a package
port installed	shows all ports currently installed
port uninstalled	shows all ports not currently installed, you may want to pipe the output of this command to grep to limit the output
port search regex	search for a port whose name matches the regex
port info package	shows meta information on a package
port list	shows the latest version of all available ports

See the port man page for additional options.

Filling in the holes

MacPorts can sometimes provide a software solution the default Apple software does not. For me, that usually means a command line or system oriented tool, but there are plenty of GUI applications available, too. The MacPorts team has done a nice job "porting" the ports system to the Mac. It is one of the most popular features of BSD, and one of the reasons I am running a Mac today.

Learn to Talk AWK

2008-12-13T11:30:00.001-08:00

When it comes to slicing and dicing text, few tools are as powerful, or as underutilized, as awk. The name AWK was coined from the initials of it's authors, Aho, Weinberger, and Kernighan. Yes, the same Kernighan of the famous K&R C Programming Language book. In the Linux world, every distribution includes the GNU version, gawk, (/bin/awk is usually a symbolic link to /bin/gawk). OS X ships with the BSD version of awk, more closely related to the original UNIX awk. The focus of the article is on the core features common among POSIX-compliant awks.

note: originally published January 16, 2006 on linux.com
updated for OS X on October 2, 2007

Basic command line usage

The awk utility is a small program that executes awk language scripts, often one-liners, but just as adeptly larger programs saved in a text file. For example, to execute an awk script saved in the file prg1.awk, and have it process the file, data1, you could use this command:

awk -f prg1.awk data1

The result is written to standard out, so it is usually piped to a result file.

Another command line parameter commonly used is -F to change the default field separator of a blank space. The field separator can also be changed within an awk program. To tell awk how to split data into fields from a comma separated variable (CSV) file, you would use:

awk -F"," -f prg1.awk data1

You may also include more than one data file to process and awk will keep running until it runs out of data:

awk -F"," -f prg1.awk data1 data2 data3 data4 data5

If you want to assign a value to a variable before execution of the program, use the -v option:

awk -v AMOUNT=100 prg1.awk data1

Behold the power

The power of awk comes from how much it does automatically for you when crunching text files, and from the simple elegance of the language. When you feed awk a text file, it does the following things for you:

Opens and reads all input files listed on the command line
Handles memory management for all variables
Parses each line and splits it into fields using the field separator (the default is a blank space, but can be changed)
Presents each line of text to your program as variable $0
Presents each field from each line in predefined variables, starting with $1, $2, ... $N
Maintains many internal variables for your use such as (but not limited to ):
- RS = record separator
- FS = field separator
- NF = number of fields in the current record
- NR = number of records processed so far
Automatically handles conversion between internal data types (string, floating point, array)
Executes the BEGIN block before processing any records (a good place to initialize variables)
Executes the END block after processing all records (a good place to calculate report totals)
Closes all input files listed on the command line

The awk language uses only three internal data types: strings, floating point numbers, and arrays. Variables do not have to be defined before they are used. Awk handles converting data from one type to another as necessary. If you add two strings together using the addition operator (+) and they contain numeric values, you get a numeric result. If a string is used in an arithmetic operation but can't be converted to a number, it is converted to zero. Usually, awk does what you want when handling data conversion.

It can open, read, and write to more files than those listed on the command line by using the getline function or redirecting output from within a program. It has access to a set of internal functions that include math, string manipulation, formatted printing (similar to the C language printf), and miscellaneous functions like pseudo-random numbers. You can also create your own functions or function libraries that can be used in several programs. All of this is packed into an executable usually about 500k in size. Programmers can typically become proficient in awk within a day. Complete references are available in a single book. You don't need a "bookshelf" of dead trees and CDs to master awk.

Implementations or ports of awk are available on nearly every platform, making your scripts reasonably portable.

AWK in the real world

Here is a short example of an awk application I created to import a list of email addresses and names from Novell Groupwise to PHPList (a mailing list manager). The list was exported from Groupwise in vCard File format (VCF), a text based format. Here is an example entry from the VCF file:

BEGIN:VCARD
VERSION:2.1
X-GWTYPE:USER
FN:Bar, Foo
ORG:;GREEN
EMAIL;WORK;PREF:foobar@yahoo.com
N:Bar;Foo
X-GWUSERID:foobar
X-GWADDRFMT:0
X-GWIDOMAIN:yahoo.com
X-GWTARGET:TO
END:VCARD

The target format was a CSV file that PHPList could import into an existing mailing list. I needed to extract the name (from the record that starts with "FN" and the email address from the record that starts with "EMAIL". These records are easy to identify and a small awk script does the job nicely.

I started construction of the script by setting up a custom record separator and a block of code to handle each record type. I saved the script in a text file called extract-emails.awk. Note that the .awk file extension is just convention, the file containing awk commands can be named anything. This was the beginning of the script:

BEGIN { FS = ":" } /^FN/ { # handle name here } /^EMAIL/ { # handle email address here }

The BEGIN block is run once before any records are read. It sets the field separator to a colon so awk will split the fields of the file when a colon is encountered.

The regular expressions /^FN/ and /^EMAIL/ tell awk to look for the characters "FN" or "EMAIL" at the start of a record, and if a match is found, run the associated block of code between the curly braces. This kind of regular expression match is common in awk but not required. A block of code with no match expression is run for every record processed by awk. I added a couple of comments (lines starting with "#") to document what each part of the script does.

Looking at the VCF data, I noticed that the "FN" record always precedes the "EMAIL" record, so I ordered the code blocks to process the records that way. Awk reads and executes a script in the order it appears. Many times, the order of the code will not matter, but in this case it does. The name is related to the email and I need to retain that relationship as the file is read, so I saved the name in an internal variable, then wrote both the email address and name to standard out while processing the email record.

Getting back to the task, let's complete the name section. The goal is to reformat the name from "lastname, firstname" into "firstname lastname", removing the comma. Here was my solution for the name:

/^FN/ { # handle name here fullname = tolower($2) split(fullname, names, ",") name = names[2] names[1] }

Knowing that awk has split up the incoming records into fields using a colon as the field separator, the field variables for the example "FN" record contain the following:

 $1 = "FN"
$2 = "Bar, Foo"

Working with the $2 variable, I used a built in awk function, towlower(), to convert the names to lowercase and stored the result in a variable called "fullname". Next, I used the split function to break the name into first and last name parts, with the result stored in an array called "names". Finally, I glued the name back together in the desired order, without the comma, and stored that result in a variable called "name".

There is very little to do inside the email code block. Awk provides the email address to us in the $2 variable (note that $2 in the "EMAIL" record is different than $2 in the "FN" record). For consistency, I converted it to lowercase, then used the print function to write both the email address and name to standard out, with a comma separating the values. Here is the complete script:

BEGIN { FS = ":" } /^FN/ { # handle name here fullname = tolower($2) split(fullname, names, ",") name = names[2] names[1] } /^EMAIL/ { # handle email address here mail = tolower($2) print mail "," name }

A sprinkle of shell glue

To pull it all together, we need a little shell glue. A small shell script allows us to call awk with the command line parameters we want and to easily redirect the output to a file. It is also handy to run a shell script when you are testing.

#!/bin/sh # Extract e-mail addresses from VCF file for PHPList. awk -f extract-emails.awk groupwise.vcf > phplist-emails.txt

Awk can be used as an intermediate step in a larger shell script where the output is fed into another utility such as sort, grep, or another awk script.

Finally, here is a sample of the output:

foobar@yahoo.com, foo bar
barbaz@yahoo.com, bar baz

Where AWK falls short

There are certain tasks that are beyond the capabilities of awk. For instance, if you need to do anything that communicates using network sockets, awk is not your best bet. Similarly, if you need to process binary files, awk falls short. The latest version of GNU awk does have some rudimentary network capabilities, but Perl, PHP, and Ruby are much better equipped for those tasks.

A million household uses

Awk is an expert tool for text processing, and the roots of Perl are clear in it's design. It is powerful enough to handle almost any kind of text crunching or reporting, while being very easy to learn and use. There is a lot of competition and many choices when it comes to scripting languages, but I still find awk the best choice for many problems. Although awk is employed most often for smaller problems, it can be used for large applications. I worked on a 12,000 line awk application used to adjudicate dental claims. This application was the core system for a successful million dollar business. Despite being a common utility on every Linux system, awk remains relatively obscure. If you take the time to learn it, the rewards will last a lifetime.

Cron fields and crontabs

2008-12-13T10:56:00.000-08:00

Crontab fields

Here is a typical crontab entry and the definition of cron fields:
00 03 * * * /tmp/zing.sh

minute
hour
day of the month
month of the year
day of the week
program or command to run

An asterisk (*) in any field means run every time for this field.
Ranges (X-Y) and steps (X-Y/Z) can also be defined.

User crontabs (including the root user)

To edit a user crontab (including root):
crontab -e

To delete a crontab:
crontab -r

System crontab

The system crontab is stored in /etc/crontab. It can be changed (as root) with a text editor.

The system crontab has one extra field before the program to run, which is the user to run the command as (usually root).

Periodic jobs in OS X Tiger

The system cron jobs were replaced in Tiger with launch daemons, in /System/Library/LaunchDaemons/. There are daily, weekly, and monthly launch daemons for system maintenance. They run shell scripts in /etc/periodic/.

The system shell scripts will run user scripts on the same schedule if they exist. To take advantage of this, simply create your own local scripts called /etc/daily.local, /etc/weekly.local, or /etc/monthly.local. They are run as root.

Spotlight on the command line

2008-12-13T09:04:00.000-08:00

In addition to the traditional command line search programs, locate and find, OS X provides a command line interface to the Spotlight search system (mdfind).

While I can't find it documented anywhere, my guess is that "mdfind" stands for "meta data find".

Simple spotlight search

mdfind searchterm

Search in a specific directory

mdfind -onlyin /Users/keithw searchterm

Search for specific meta data

mdfind "kMDItemAuthor == '*Donaldson*'"

List available meta data fields

mdls filename

Turn off spotlight indexing for a volume

mdutil -i off volumename

Turn on spotlight indexing for a volume

mdutil -i on volumename

Show the status of spotlight indexing for a volume

mdutil -s volumename

How spotlight does its magic

Unlike the locate program and database, spotlight recognizes and indexes file system changes almost immediately. How is that possible? The magic happens in the kernel, which sends notices of file system changes to a special character device named /dev/fsevents. Spotlight monitors this special device and learns about new files that need to be indexed. There is no published API for /dev/fsevents, as far as I can tell.

Surfing the Keyboard in BASH

2008-12-13T08:58:00.001-08:00

You are probably familiar with common keyboard sequences like Ctrl-C to end a program, but there are dozens of useful shortcuts you can use in BASH to edit the command line, move around your command history, and control jobs. I've collected 18 useful Ctrl and Meta (Option) sequences to make you more productive in BASH. So grab your 'board and leave the wet suit at home, the water is warm this time of year.

Command Line Editing

BASH keyboard sequences come in two flavors, Ctrl and Meta. Ctrl sequences are straightforward, you hold down the Ctrl key while pressing another key. On a PC with Linux, the Meta key is usually mapped to Alt, but on a Mac, it is usually mapped to the Esc key. Since the Esc key is poorly placed for serious keyboarding, I suggest mapping the Meta key to the Option key.

To map Meta to Option in the OS X Terminal, use the menu and go to Terminal / Window Settings, select Keyboard in the drop down box at the top, and check 'Use option key as meta key'.

To map Meta to Option in iTerm, use the menu and go to Bookmarks / Manage Profile, expand Keyboard Profiles, select Global, and check the 'Option Key as Meta' radio button.

With that out of the way, here are some shortcuts to edit commands.

Key Sequence	Result
Ctrl-A	moves the cursor to the beginning of the line
Ctrl-E	moves the cursor to the end of the line
Ctrl-D	deletes the character under the cursor
Ctrl-F	moves the cursor forward one character
Ctrl-B	moves the cursor backward one character
Ctrl-K	deletes from the cursor to the end of the line
Option-B	moves the cursor back one word
Option-F	moves the cursor forward one word
Option-L	downcase a word starting from the cursor
Option-U	upcase a word starting from the cursor

Moving through history

BASH records all commands you enter in a personal history file called .bash_history in your home directory. Notice the dot (.) at the start of the name, indicating it is a hidden file. Finder won't show you hidden files without some tweaking, but you can easily see hidden files in Terminal with the ls -a command. If you are paranoid, BASH has options to turn history off, but it is very useful if you work much at the command line. Here are the history shortcuts:

Key Sequence	Result
Ctrl-R	start an incremental reverse history search, Ctrl-J to stop
Ctrl-S	start an incremental forward history search, Ctrl-J to stop
Ctrl-J	stop incremental search, leaving command to edit
Ctrl-P (also up arrow)	fetch the previous command from history
Ctrl-N (also down arrow)	fetch the next command from history

Incremental searches work by matching the string entered with the most recent history command that contains the string. The more you type, the more accurate the match. Use Ctrl-J (or the left or right arrow keys) to end the search, leaving the matched command on the command line ready to either edit or run.

Job Control

OK, nothing new here...

Key Sequence	Result
Ctrl-Z	suspends the foreground job, returns control to user
Ctrl-C	send signal 3 (SIGQUIT) to the foreground job

Miscellaneous

Key Sequence	Result
Ctrl-X Ctrl-V	show current version of BASH

All examples given above are default key bindings in BASH. There are additional BASH functions that are not automatically bound to a key sequence. To see all current key bindings, plus all unmapped functions, run this command:

bind -P

If you want to set your own key sequences or macros, you can define them in a readline run control file in your home directory called .inputrc (you will need to create one). Readline is a library that provides command line control for BASH. To define a new key sequence, edit the .inputrc file and enter your new definitions one per line. You need to use the readline syntax as follows:

key-sequence: function

\C- means Ctrl plus another key
\M- means Meta (Option) plus another key
\e is the ESC character
\\ is a backslash
\" is a double quote
\' is a single quote

For example, to bind Option-z (meta-z) to the tty-status function, add this line to your .inputrc file:

"\M-z": tty-status

Close Terminal and start a new shell so it reads the .inputrc file, then try Option-z and you should see something like this:

load: 0.61 cmd: bash 10561 running 0.00u 0.00s

You can create your own key sequences for anything BASH supports, or change them to taste. Few people have a burning desire to change the default BASH key sequences, but if you need to change your keyboard surfing style, you know how... dude.

PostgreSQL

2008-12-13T08:57:00.000-08:00

PostgreSQL is a robust and powerful open source database. It has more advanced features than any other open source database and scales well with huge datasets and high traffic loads.

By default, PostgreSQL listens on TCP port 5432.

It is not installed by default in OS X.

Dump all databases

pg_dumpall --clean > databases.sql

Dump a database with compression (-Fc)

pg_dump -Fc --file=database.sql --clean database

Dump a database, plain text, one schema only (-n)
pg_dump -Fp --file=filename.sql -n schema --clean database

Dump a single table

Specify the schema with the table name (if applicable) with pg_dump [-d database] -t schema.table

Dump a table definition (no data)

pg_dump -s [-d database] -t schema.table

Restore a database from a dump file

pg_restore -Fc database.sql

Restore a single table from a dump file

pg_restore -v -e -Ft -d database -t tablename dumpfile.tar
note: in this case, the dump file is in tar format, the database to restore to is after the -d switch and the table to restore is after the -t switch.

Copy data from a file into a table (from the psql client)

COPY table-name FROM '/path/to/filename' DELIMITER 'delimiter';

note: the file must be readable by postgresql (make it 755),

the default delimiter is tab.

Copy data from a table to a file (from the psql client)

COPY table-name TO '/path/to/filename' DELIMITER 'delimiter';

note: the directory and file must be writable by postgresql,

the default delimiter is tab

Start the PostgreSQL interactive terminal

psql

Psql - show a list of databases

\l
Lowercase L, not the number 1

Psql - show all users

select * from pg_user;

Psql - show all tables (including system tables)

select * from pg_tables;

Psql - show tables in the current context (database/schema)

\d

Psql - show description of tablename

\d tablename

Psql - show description of tablename, along with constraints, rules, and triggers

\d+ tablename

Psql - change current database

\c database;

Psql - show all schemas in the current database

\dn

Psql - Grant permissions on a schema to a user

GRANT ALL ON myschema TO user;

Psql - quit psql

\q

Psql - show help

\?

Psql - copy a table to a tab delimeted file

COPY table TO 'table.txt';

Psql - load a table from a tab delimeted file

COPY table FROM 'table.txt';

Psql - show permissions on database objects

\z [object]

 r -- SELECT ("read")

w -- UPDATE ("write")

a -- INSERT ("append")

d -- DELETE

R -- RULE

x -- REFERENCES (foreign keys)

t -- TRIGGER

X -- EXECUTE

U -- USAGE

C -- CREATE

T -- TEMPORARY

arwdRxt -- ALL PRIVILEGES (for tables)

* -- grant option for preceding privilege

/yyyy -- user who granted this privilege

Psql - getting or setting sequence values
Get current value of a sequence:
SELECT currval('this_id_seq');

Set current value of a sequence to 1000:
SELECT setval('this_id_seq', 1000);

Run the vacuum utility

vacuumdb --verbose --analyze --all
Note: vacuum reclaims space from deleted records and updates indexes. It should be set up in cron. Newer versions of postgresql may run vacuum automatically.

Increase perfomance with shared memory

One effective performance tuning tip for Postgresql is to increase the shared memory buffers. This might require adding RAM to the server. Many Linux distros default to 32MB of shared memory, controlled by two kernel parameters:
/proc/sys/kernel/shmmax /proc/sys/kernel/shmall

These values can be changed at run time, but it is better to set them at boot using the /etc/sysctl.conf file. This increases shared memory to 1GB:
# increase shared buffers for postgres at boot kernel.shmmax=1073741824 kernel.shmall=2097152

Then, tell PostgreSQL to use 768MB of the 1GB available in the /var/lib/pgsql/data/postgresql.conf file:
shared_buffers = 98304 # min 16, at least max_connections*2, 8KB each

Restart PostgreSQL for the change to take effect.

MySQL

2008-12-13T08:56:00.001-08:00

MySQL communicates through either local unix sockets or over TCP/IP port 3306 (default). Database names, tables, field names, and passwords are case sensitive. SQL Commands are not case sensitive.

The main configuration file is /etc/my.cnf. Usually doesn't need tweaking, except when using the InnoDB storage engine, or if you have high performance requirements.

The main command line utilities are mysql, mysqldump, and mysqladmin. Many people like the phpMyAdmin package to manage MySQL through a web browser.

MySQL is not part of the default install in OS X.

Server Administration

Show all running MySQL processes
mysqladmin --user=root --password=xxx processlist

Show detailed status report
mysqladmin --user=root --password=xxx extended-status

Reload grant tables (after making security table changes)
mysqladmin --user=root --password=xxx reload

Show running configuration settings
mysqladmin --user=root --password=xxx variables

Kill a slow or locked process
First, get the process id using processlist, then
mysqladmin --user=root --password=xxx kill id

Reset the value of an autoincrement field in a table

mysql --user=root database
alter table tablename autoincrement=100;

Note: the above resets the autoincrement field to 100. Use caution!

Alter a table so it can grow beyond 4 GB in size

Even if your OS supports file sizes greater than 4 GB, and even though MySQL supports tables larger than 4 GB, it will still give you a "table full" error message if you try to insert records into a table that has reached 4 GB unless you tell it to allow tables to grow larger. You can set a global option in my.cnf or you can tell MySQL on a table by table basis. This command will allow a table to grow to roughly 300 GB:

ALTER TABLE tbl_name MAX_ROWS=1000000000 AVG_ROW_LENGTH=300;

Security

Change/set the root password

mysql --user=root mysql (initially no password)
update user set Password=password('new_password') where user='root';
flush privileges;

Create a user with remote update authority

mysql --user=root --password=xxx mysql
insert into user (Host, User, Password, Select_priv, Insert_priv, Update_priv, Delete_priv) values ('%', 'remote', password('xxx'), 'Y', 'Y', 'Y', 'Y');
flush privileges;

Create a user with access to just the db1 database

mysql --user=root --password=xxx mysql
insert into user (Host, User, Password) values ('localhost', 'foo', password('xxx'));
insert into db (Host, Db, User, Select_priv, Insert_priv, Update_priv, Delete_priv) values ('localhost', 'db1', 'foo', 'Y', 'Y', 'Y', 'Y');
flush privileges;

Backup and Restore

Dump all databases (schema and data)
mysqldump --user=root --password=xxx --all-databases > databases.sql

Dump a single database (schema and data)
mysqldump --user=root --password=xxx --databases db1 > db1.sql

Dump a single database (schema only)
mysqldump --all --no-data --user=root --password=xxx --databases db1 > db1.sql

Restore a database from a dump file

mysql --user=root --password=xxx <>

Load data from a text file
  The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed. 
  For security reasons, when reading text files located on the server, the files must either reside in the database directory or be readable by all. Also, to use LOAD DATA INFILE on server files, you must have the FILE privilege on the server host. You can also load datafiles by using the mysqlimport utility; it operates by sending a LOAD DATA INFILE command to the server. 
  The defaults cause LOAD DATA INFILE to act as follows when reading input: 
Look for line boundaries at newlines.
Do not skip over any line prefix.
Break lines into fields at tabs.
Do not expect fields to be enclosed within any quoting characters.
Interpret occurrences of tab, newline, or .. preceded by .. as literal characters that are part of field values.
 
Example using the defaults (tab delimited):
LOAD DATA INFILE "data.txt" INTO TABLE table;

Example using an unquoted CSV file:
LOAD DATA INFILE "data.txt" INTO TABLE table FIELDS TERMINATED BY ',';

Tuning

Log slow queries
 MySQL can log queries that take a long time to complete by adding this option to /etc/my.cnf in the [mysqld] section:
log-slow-queries=/var/log/mysqld.slowquery.log

The file should be created first with an owner of mysql:mysql. By default, a slow query is one that takes more than 10 seconds.



Gnu Privacy Guard (GPG)
2008-12-11T17:05:00.001-08:00
The Gnu Privacy Guard (GPG) is a command line application that can use public key encryption to safeguard text and data and prepare it for transport through the email system.

It is not part of the default install of OS X and must be installed manually.

To generate a new key pair:
gpg --gen-key

To list keys on your public keyring:
gpg --list-keys

To list keys on your secret keyring:
gpg --list-secret-keys

To encrypt a text file "message.txt" for recipient "foo" with ASCII armor (Base64):
gpg -e -a -r foo message.txt
The encrypted message is saved as file "message.txt.asc".

To encrypt a text file "message.txt" for recipient "foo" with ASCII armor and sign it with your secret key:
gpg -s -e -a -r foo message.txt

To import a public key:
gpg --import keyfile

To sign a newly imported key with your secret key:
gpg --sign-key keyname

To delete a public key from the keyring:
gpg --delete-key keyname

To verify a file with a detached signature:
gpg --verify signature data-file

To import a public key:
gpg --import keyfile

Set up a trusted public key (no passphrase required):
 If you want to encrypt files in a script and not be prompted for your passphrase, you need to sign all public keys you want to use. Follow this procedure (GPG 1.2.5+) to sign a public key. 
gpg --edit-key keyname
at the prompt, enter "trust"
select "4" for trust fully
enter "lsign" to locally sign it
at the prompt, enter 3 for very careful checking
answer "yes" to the the "Really Sign?" prompt
enter secret key passphrase when requested
enter "save"
 The key is now signed and can be used in a script without passphrase requirements.


A Bourne Again Daemon
2008-12-11T16:58:00.000-08:00
In Unix-like systems, a daemon is a process that runs in the background, usually providing some kind of service. Most daemons can be identified in the process list because their names end in d. Examples are the Apache web server (httpd) and the OpenSSH server (sshd). Another thing daemons usually have in common is that they are written in the C programming language. To create a daemon quickly, you can throw one together using the Bourne Again Shell (Bash) scripting language.

note: originally published April 20, 2006 on linuxboxadmin.com
updated for OS X on August 15, 2007


More about daemons
 There are a couple of other common daemon characteristics. They often have a configuration file with a name ending in ".conf". You can tell a daemon to read its configuration file again by sending it signal HUP (1). Lastly, daemons usually listen on a particular TCP or UDP port for incoming network connections. The network functionality is the only feature we won't duplicate using BASH. It is possible to add a network aware program to a BASH daemon, but that is left as an exercise for the reader. 
 Please allow me to introduce myself
 Here is the basic structure of a BASH daemon: 
#!/bin/bash
# This is a simple daemon shell script.
#
# Traps:
# signal HUP (1) read config file again
# signal QUIT (3) delete temp files
# signal TERM (15) delete temp files

conf=/tmp/bashdaemon.conf
source $conf
tempfile=/tmp/BASHDAEMON_$$
touch $tempfile

# set traps
trap 'source $conf' 1
trap 'rm -f $tempfile; exit' 0
trap 'rm -f $tempfile; exit' 3
trap 'rm -f $tempfile; exit' 15

# loop forever
while :
do
   echo $foo
   echo $bar
   sleep 30
done
 If you want to play along at home, save the script as bashdaemon.sh and make it executable with:
chmod a+x bashdaemon.sh
  The bashdaemon.conf file contains variable assignments for two variables, foo and bar.  It starts out like this:
foo=famine
bar=death
    Let's walk through the script line by line. 
  conf=/tmp/bashdaemon.conf

This line defines the configuration file, called /tmp/bashdaemon.conf. In the example, all files are in the /tmp directory, but if you want to follow daemon convention, you could put it in /etc. 
  source $conf

The source command is a BASH built-in that reads and executes the configuration file, activating the variable assignments. 
  tempfile=/tmp/BASHDAEMON_$$
touch $tempfile

The tempfile serves two purposes. It can be used as work area and it stores the process ID of the daemon in the file name. For example, if the process ID was 1234, the tempfile name would be /tmp/BASHDAEMON_1234. As long as the daemon is running, the tempfile will exist. It gets automatically deleted when the daemon ends (unless it is killed with signal KILL (9)). 
  trap 'source $conf' 1

This trap allows the daemon to capture signal HUP (1) and tells it to source the configuration file again and continue running. If the variable assignments have been changed, the new values take effect. 
   trap 'rm -f $tempfile; exit' 0
trap 'rm -f $tempfile; exit' 3
trap 'rm -f $tempfile; exit' 15

These lines trap signals 0, 3, and 15 to delete the tempfile, then exit. See below for more about traps. 
 while :
do
   echo $foo
   echo $bar
   sleep 30
done
  The loop uses a never ending while statement to run until it is ended with one of the signals that tell it to exit or signal 9, killing it unconditionally. Inside the loop, the daemon does its work. In the example, it just prints the contents of the foo and bar variables, then goes to sleep for 30 seconds. 
 Traps
 For those not familiar with traps, a few more words are in order. If you already know about traps, skip ahead. Linux (and Unix) defines a set of messages called signals that can be sent to a program to tell it what to do. The "kill" program is used to send signals from the command line. To view the complete list of signals, try:

kill -l

That is an L, not the numeral one. For example, the hang up signal (HUP) is signal 1 and the QUIT signal is signal 3. When a program receives the QUIT signal, it exits immediately, unless the program is written to recognize that signal and take other action. Intercepting a signal is called a trap. Most languages have a trap function and in BASH, it is the trap built-in function. The one signal that can't be trapped is KILL -- signal 9. The man page has more details. 
  Firing it up
 There are many ways to run a Bash daemon, depending on how formal you want to get. The least formal way is to start it from a terminal session with the background operator:

./bashdaemon.sh &

This starts the daemon in the background and works, but requires the terminal session to remain open. Another option is to start it from a screen session. Screen is a program that lets you detach a process from the terminal and reattach to it later, even from a different computer. If you haven't tried screen before, it is a powerful tool worth learning. After launching the daemon from screen, you can simply detach, close the terminal, and check on it later as needed. 
  The most formal option in OS X is to set up a property list file like the big daemons use and have launchd manage it. Linux users could set up a run control script in /etc/rc.d/init.d/. On the other hand, if you need to go to these lengths, you probably need to write it in a more appropriate language. BASH daemons excel at quick and dirty solutions. 
  Killing it softly
To test the signal HUP trap, I started the BASH daemon in the background, then changed the variable assignments in the bashdaemon.conf file. After my edits, the bashdaemon.conf file looked like this:
foo=pestilence
bar=war
   Next, I found the process ID of the daemon (1927) by looking at the tempfile name in /tmp.  Then, I sent signal 1 to it using:

kill -s HUP 1927

After it received the signal, it started printing the new values from bashdaemon.conf. To kill it for good, I pulled the daemon back into the foreground with code>fg and used Ctrl-C to end it. Afterward, the tempfile was gone as expected. 
  Daemons for work and play
Some interesting near real-time stuff can be done with BASH daemons. You could monitor a database, be notified when a certain user logs in, take action when a file is accessed, when a program is run, or anything you dream up. Summon your own daemons to do your bidding. 



Perl
2008-12-11T16:56:00.000-08:00
Perl is a popular text processing language that has been extended into nearly every corner of computing. It is not one of my favorite scripting languages, but there are some things it does very well. I sometimes run across Perl applications that I want to run or customize. These tips can be useful in managing Perl.

Perl is part of the default install of OS X.

Set up CPAN

perl -MCPAN -e shell

Find all Perl modules currently installed

find `perl -e 'print "@INC"'` -name '*.pm' -print

Taint mode

For untrusted scripts, add -T flag to perl at the start of the script:
#!/usr/local/bin/perl -T

One liner to edit files in place

This command edits a group of files in place.  In particular, it does a global search/replace.  The -p switch tells perl to iterate over filename arguments, -i  tells perl to edit files in place, and -e indicates that a one line program follows.

perl -p -i -e 's/248/949/g' *.txt

One liner to edit files in place and backup each one to *.old

perl -p -i.old -e 's/248/949/g' *.txt


Rsync
2008-12-11T16:52:00.001-08:00
Rsync is a file and directory syncronization tool. It can be used to keep local and/or remote files syncronized. What makes rsync powerful is that it can detect changes within large files and only transfer the parts that are different. This is much more efficient than transfering entire files, especially as the file sizes get bigger.

The rsync utility is part of the default installation in OS X.

Note: examples that use a shell use ssh with public key authentication

To synchronize a local directory with a remote one (pull), use:

rsync -r -a -v -e "ssh -l username" --delete hostname:/remote/dir/ /local/dir/

To synchronize a remote directory with a local one (push), use:

rsync -r -a -v -e "ssh -l username" --delete /local/dir/  hostname:/remote/dir/

To synchronize a local file with a remote one, use:

rsync -a -v -e "ssh -l username" hostname:/filename  /local/filename

To synchronize a remote file with a local one, use:

rsync -a -v -e "ssh -l username" /local/filename hostname:/filename

To synchronize a local directory with a remote rsync server:

rsync -r -a -v --delete rsync://rsync-server.com/stage/ /home/stage/

To synchronize a local directory with a local directory (make a backup), use:

rsync -r -a -v --delete /local/dir/ /backup/dir/


BASH job control: fg, bg, jobs, and Ctrl-Z
2008-12-11T16:51:00.000-08:00
The BASH shell has a feature called job control that allows you to run and manage multiple processes from a single interactive prompt. In the age of serial connected dumb terminals, this was a killer feature. In the age of multi-homed multi-core Macs, not so killer, but there are certain situations where it can still be useful.

The commands I want to focus on are bg (background), fg (foreground), and jobs. These are all shell builtins, meaning they are commands built into and handled by BASH itself instead of being external programs, like the top program. Examples in this article were run on Mac OS 10.4.10.

Since the advent of the GUI, one solution to running multiple shell commands at a time is to open more than one terminal window. The beauty of this solution is that you can see both display windows at the same time and monitor what is happening. You can also copy and paste information between them. 
 What about times when you are connected to remote Mac (or Linux or Unix) computer with secure shell? Now, you are back to the single interactive shell and the job control commands can be of assistance. Yes, you can always open another terminal on your computer and establish a second secure shell connection, but that may be more trouble than it's worth. 
 Here is a scenario where job control can help get your work done a little faster. Again, say you have logged into a remote computer using secure shell and you start a long running job, such as compiling a large program from source, or updating the locate database. As an aside, the locate database came to the Mac from the BSD world where there was no spotlight. It lets you quickly find files based on a partial file name. It doesn't search inside files like spotlight, it only looks at file names. The Mac command for updating the locate database is a C shell script named /usr/libexec/locate.updatedb. It can be run directly:

keithw$ /usr/libexec/locate.updatedb

note: if run as a regular user, it won't be able to scan all directories, so it should really be run as root. 
  Since it takes a long time to run, you will be unable to run any other commands until it finishes. The script is running in the foreground of the interactive shell. You can force it into the background by using Ctrl-Z. The Ctrl-Z sequence pushes the foreground job into the background, pauses it, and gives control back to you. When I use Ctrl-Z, I see this output:

[1]+  Stopped      /usr/libexec/locate.updatedb

This means job number 1 has been stopped and shows the associated command. To check the status of the all background jobs, use the jobs command with the -l (list) switch:

keithw$ jobs -l
[1]+  4836 Suspended               /usr/libexec/locate.updatedb
  Now we have control of the terminal back, but the script we started is currently not running.  That can be fixed by using the bg command. The bg command tells BASH to run the specified job number in the background. You don't need to give it a job number if there is only one background job. This gets the script running again:

keithw$ bg
[1]+ /usr/libexec/locate.updatedb &
keithw$ jobs -l
[1]+  4836 Running                      /usr/libexec/locate.updatedb & 
  At this point, we have interactive control, the script is running in the background, and we can do something else while it finishes. When it is done, the shell will send us a notice. If you want to bring it back to the foreground, use the fg command. Then, you are back where you started, waiting for it to finish. Generally, you would only use the fg command, if you have multiple background jobs running, and want to interact with one of them. You could theoretically have a text editor, a compile, and a long running script all running as jobs and swap them in and out as needed. 
 Take a closer look at the output from the jobs command above. Notice the & at the end of the command? You can start a job running in the background by appending an & to the end of the command. Had we done that originally, it would have started in the background and returned interactive control immediately. But for those times when you forget, the job control commands are your friends. 
 The difference between job number and process ID I want to make a distinction between jobs and processes. Each running program in the operating system has a process ID (pid). From the shell's point of view, each background process is a job and jobs get assigned numbers starting at 1. The second background process would be job 2. To reference a background process when using the fg or bg commands, use the job number instead of the process ID. For sending operating system signals to a background process, you use the process ID. Going back to the last jobs command, the process ID is printed after the job number, in this case 4836. To kill the background process unconditionally, you would send it the KILL signal using the process ID instead of job number:

keithw$ kill -9 4836 



Tar
2008-12-11T16:50:00.000-08:00
Tar, (short for tape archiver), is a versital tool that can be used for archiving files to disk or any other device as easily as tape. In fact, if you don't work in a data center, you will probably never use tar with a tape drive.

Often, Unix/BSD/Linux files and source code are distributed in a zipped tar file, sometimes called a tarball. Extensions for tarballs are usually .tgz or .tar.gz (gz because it was compressed using gzip, the free GNU zip program). Rarely, you may run across a tar file that is not compressed and has an extension of simply .tar.

 Here are some common uses for tar. If you pass a directory or a wildcard to tar, it will include all subdirectories in the tar file by default. 
  Create a gzipped tar archive
tar czvf backup.tgz files-to-backup

Create a gzipped tar archive, preserving file permissions
tar czvpf backup.tgz files-to-backup

Extract a gzipped tar archive
tar xzvf backup.tgz files-to-backup

Create a bzipped tar archive (using bzip compression instead of gzip)
tar cjvf backup.bz2 files-to-backup

Extract a bzipped tar archive
tar xjvf backup.bz2 files-to-backup

List files in a tar archive without extracing
tar tf backup.tgz

List files in a zipped tar archive without extracing
tar tzf backup.tgz


SSH
2008-12-11T16:48:00.002-08:00
The following are tips for both the Secure Shell daemon (server) and the SSH command line client inlcuded with Mac OS X.

SSH daemon (server)
 The first time sshd runs, it generates three cryptographic key pairs and stores the keys in the /private/etc/ directory. 
ssh_host_key and ssh_host_key.pub (v1)
ssh_host_dsa_key and ssh_host_dsa_key.pub (v2 DSA)
ssh_host_rsa_key and ssh_host_rsa_key.pub (v2 RSA)
   SSH communicates over TCP port 22 by default. The global server configuration file is:
/private/etc/sshd_config 
 To deny all remote SSH root logins (generally a good idea), set this value in the sshd_config file:
PermitRootLogin no   To disable the less secure v1 SSH protocol, set this value in the sshd_config file:
Protocol 2 
  To disable X forwading, set this value in the sshd_config file:
X11Forwarding no 
  To disable keyboard password logins (force public/private key authentication), set this value in the sshd_config file:
PasswordAuthentication no 
  SSH client
 Note: because of its sensitive nature, the $HOME/.ssh/ directory and most of the files in it MUST be read/write for the user and not accessible to group or other. For example, the file permissions should look like this:
-rw-------
Otherwise, SSH will ignore them.  If you copy personal SSH files to a new system and they don't work, check the permissions. 
  The default client configuration file is:
/private/etc/ssh_config
The user configuration file, $HOME/.ssh/config takes precedence over the default configuration. 
  To connect to an SSH server using a different user ID:
ssh userid@server-name-or-IP 
  To securely copy a local file(s) to a remote server, use scp:
scp localfile userid@server-name-or-IP:remotefile 
  To securely copy remote file(s) to the local machine, use scp:
scp userid@server-name-or-IP:remotefile localfile 
  Using public/private key encryption for authentication
 First, generate a keypair for logins without passwords:
ssh-keygen -t dsa
The system will prompt you for a secret key passphrase, then create the key pair in two files:
 id_dsa (v2 private key)
id_dsa.pub (v2 public key)
 Next, append the v2 public key to your $HOME/.ssh/authorized_keys2 file on the server(s) where you want to login. 
  To bypass the passphrase that unlocks your secret key every time it is needed, load the key into ssh-agent. 
  SSH-Agent
 To load secret keys in the ssh-agent manually, execute: 
ssh-agent
ssh-add keyfile (once for each key)
   It is usually more convenient to run ssh-agent and load keys in a BASH login.  Linux users can use the keychain script. 
  Port Forwarding
 SSH can port forward local and remote connections securely. Only root can forward privileged ports (<=1024) 
 To redirect a local port to a remote host port:
ssh userid@remotehost -L localport:remotehost:remoteport 
  To redirect a remote port to a local or remote host port:
ssh userid@remotehost -R remoteport:host:localport 



Fix a broken terminal
2008-12-11T16:48:00.001-08:00
To fix a broken terminal, try one of these commands:
 tput init
tput reset
reset
[ENTER] reset [ENTER] 

If binary characters are sent to a terminal, it will often leave the screen in an abnormal state because it interprets them as control characters. A random stream of control characters can make the terminal unreadable. You may not be able to see the commands as you type them until the terminal is reset, or you may see junk characters as you type. Often, though, one of the above commands will clear up the problem.

To see the device of the current terminal:
tty

Pseudo-terminals under OS X Aqua are usually /dev/ttyp?.  For example, /dev/ttyp1, /dev/ttyp2, etc.

To see all current terminal settings:
stty -a


Daily aliases
2008-12-11T16:46:00.000-08:00
If you spend any time working with the shell, you probably use many GNU or BSD utilities. One thing that distinguishes the newer free software versions from the classic Unix versions is that the free software programs are rife with additional options. Some of these options are so useful you may want to create an alias so you can use them all the time.

note: originally published December 12, 2005 on newsforge.com
updated for Mac OS X on July 27, 2007

 A shell alias is a simple way to create your own custom command. To make your aliases available every time you open a BASH shell, add them to your $HOME/.bash_login file. For other shells, place them in the associated run control file. In OS X, the default BASH configuration doesn't include very much, really just a slightly customized prompt. When I start working on a new system, I immediately install my favorite aliases. 
  An alias is created using the following syntax:

alias name='value'

where name is the name of the new command and value is the command string that is executed. Note, if you create an alias with the same name as an existing executable program in your path, the shell will run the alias instead of the program. 
  To remove an alias from your current shell session, use:

unalias name

where name is an existing alias. 
  To see all the current aliases known to your shell, type alias with no parameters. 
  All of the following aliases were created and tested on OS 10.4.10 using the standard BASH shell. 
  List the most recently modified files and directories
 alias lt='ls -alt | head -20'

Once this alias is loaded, typing lt lists the most recently modified contents of the current directory. The a and l options show all files including hidden files in a long listing format. Since I am usually interested in a file I have just changed, I limit the output to 20 lines with the head command. 
  List only subdirectories
 alias lsd='ls -al -d * | egrep "^d"'

This shows only subdirectories of the current directory by using egrep to limit the listing to entries with the 'd' directory attribute. 
  Show the inode number for files in the current directory
 alias li='ls -ai1 | sort'

This prints the inode number (file system block identifier) for each file, sorted in ascending order. The last option to ls in this alias is the numeral one, not the letter L. You might need the inode number for file system repair or to recover data from a damaged file. 
  Scramble a file using the ROT13 algorithm  alias rot13='tr A-Za-z N-ZA-Mn-za-m'

This shifts all letters in a file 13 characters up the alphabet, creating scrambled, but not encrypted text. Running ROT13 on the scrambled text restores it. ROT13 was used back in the day on newsgroups to politely hide potentially offensive text. I almost never run across ROT13 text these days which may be a sign of lost Net innocence. To use the alias, redirect a file to standard input:

keithw$ rot13 < /etc/motd
Jrypbzr gb Qnejva! 
  Fast directory navigation with pushd and popd
 alias +='pushd .'
alias _='popd'

I tend to move around a lot on the command line. If I am working on code in a deeply nested directory, but need to go check some log files, I'll use the shell built-in pushd command to save my current directory location and popd to get back to the directory later. These aliases simplify the process by entering '+' before changing directories, and '_' to return to the directory later. You can push more than one directory on to the stack and pop them back off in reverse order. I used the underscore instead of the minus sign for popd because the minus sign is a reserved symbol. 
  Find disk space abusers
 alias dusk='du -s -k -c * | sort -rn'

Shows disk usage of files and subdirectories in the current directory, sorted with the ones using the most disk space first. The report shows disk usage in 1K increments and the total for the directory at the top. Beware, if you start this command from the root directory or at the top of a deeply nested directory, it could run a long time. Here is a sample report:
 keithw$ dusk
1756    total
528     micro
444     articles
408     images
152     move
48      cgi-bin
36      inc
28      src
24      index.html
20      index.html.new
12      xref
8       send.php
8       feed.xml
8       css
   Show all programs connected or listening on a network port
 alias nsl='netstat -alnp ip | grep -v CLOSE_WAIT | cut -c-6,21-94 | tail +2'

Since netstat is used to look at all system info, you need to run this as root. It shows the process ID and program name of everything either connected or listening on a network port, including the sockets of the sending and receiving hosts. This is a great way to make sure no unexpected services are running in the background. Here is a sample report:
Proto  Local Address          Foreign Address        (state)
tcp4   192.168.1.137.63819    209.85.139.83.443      ESTABLISHED
tcp4   192.168.1.137.63817    209.85.139.83.443      ESTABLISHED
tcp4   192.168.1.137.22       199.199.199.199.4570   ESTABLISHED
tcp4   *.*                    *.*                    CLOSED
tcp4   127.0.0.1.1033         127.0.0.1.990          ESTABLISHED
tcp4   127.0.0.1.990          127.0.0.1.1033         ESTABLISHED
tcp4   127.0.0.1.631          *.*                    LISTEN
tcp4   *.5432                 *.*                    LISTEN
tcp6   *.5432                                        *.*
   Combinations and Permutations
 If you want to dig deeper into any of the options used in these aliases, check out the man/info page for the command. There are so many permutations of options and commands that you can sometimes stumble across a cool solution by studying the more obscure options in the man/info pages. What are your favorite aliases? 



Unix permissions on OS X
2008-12-11T16:41:00.000-08:00
There are nine standard permissions in Unix-like operating systems on each file (and directory). There is a set for the owner of the file, a set for the group owner of the file, and set for everyone else. Each set has a Read (r), a Write (w), and an eXecute (x) permission. You can view the permissions on a file by using the ls (list) command with the long format option (-l). The permissions are listed in rwx order. For example:

keithw$ ls -l
-rw-r--r--   1 keithw  keithw   6677 Apr  5  2006 ab.php

The permissions are read+write for the owner, read for the group, and read for everyone else.

In addition to the standard permissions (rwx), there are three special permissions that can be set for a file or directory: suid, sgid, and sticky bit.

suid
this special permission allows the file to be executed with the security permissions of the file owner instead of the permission of the user who ran the program. This can be a source of security problems. Some daemons run as suid root. The suid permission is seen as an "S" in the user executable position a long directory listing (ls -l). Has no effect if the file is not executable.

To set the suid permission:
chmod u+s filename 
  sgid
this special permission allows the file to be run with the security permissions of the group instead of the permission of the user who ran the program. This can be a source of security problems. The sgid permission is seen as an "S" in the group executable position in a long directory listing (ls -l). Has no effect if the file is not executable.

To set the sgid permission:
chmod g+s filename 
  note: If sgid is set on a directory, any file created within that directory will have the same group owner assigned as the directory. Useful when a group of users is sharing the same directory. 
  sticky bit on a directory
Prevents any files in a directory from being deleted by anyone but the owner of that file. Often used on the /tmp directory. Good to prevent accidental deletions by rm * commands. The sticky bit is seen as a t in the other executable position in a long directory listing (ls -l). Setting the sticky bit on a file is ignored.

To set the sticky bit:
chmod u+t dirname

note: in Linux, the option is set using the "other" permissions instead of "user":
chmod o+t dirname

In both cases, the "t" appears in the other executable position:
drwxr-xr-t    2 keithw  keithw     68 Jul 26 09:02 test
  Finally, Unix permissions are not the end of the story. The OS X file system can also use Access Control Lists stored in extended attributes to give you more fine grained access control. You can view extended attributes using the -e option of the ls command.  See the chmod and ls man pages for more details. 



SMTP testing from the command line
2008-12-11T16:40:00.002-08:00
To test SMTP from the command line:
telnet host-to-test 25 (connect to port 25 on mail server)
HELO sending-host
MAIL FROM: foo@foo.com
RCPT TO: bar@bar.com
DATA
(enter one blank line after DATA)
   Subject: test
  To: to-user
  From: from-user
  (enter one blank line after From:)
  test text for email
  . (enter a single period by itself on the last line)   
QUIT

Read on for testing servers that use SMTP AUTH...

To test SMTP with SMTP AUTH (no TLS/SSL) from the command line:
 First, use a Perl one liner to compute the Base64 encoded user ID and password needed to authenticate to the mail server. This requires the Perl MIME::Base64 module.

perl -MMIME::Base64 -e 'print encode_base64("00userid00password")'

The output looks something like this:
AHVzZXJpZABwYXNzd29yZA==
Copy the output string so it can be pasted into the AUTH step below. 
 telnet host-to-test 25 (connect to port 25 on mail server)
HELO sending-host
AUTH PLAIN Base64-encoded ID/password
MAIL FROM: foo@foo.com
RCPT TO: bar@bar.com
DATA
(enter one blank line after DATA)
    Subject: test
   To: to-user
   From: from-user
   (enter one blank line after From:)
   test text for email
   . (enter a single period by itself on the last line)   
QUIT



List Open Files (lsof)
2008-12-11T16:40:00.001-08:00
The lsof program is a great tool to find open files. One case you might need this is to find a process that has an open file on a device you are trying to unmount. Another is if you want to determine what files a certain process is opening.

List your own open files (includes sockets and pipes):
lsof

List all open files on the system (includes sockets and pipes), must be root:
lsof

List open files for certain processes, must be root:
lsof -c pattern
where pattern is the first few letters of the process name or a regular expression between forward slashes /regexp/.

List all open network sockets, must be root:
lsof -i
Useful to make sure something unexpected isn't listening to the network.


Vi[m]
2008-12-11T16:38:00.000-08:00
Vi is an one of two powerhouse text editors in the Unix world, the other being EMACS. While obtuse, vi is extremely powerful and efficient. There may be times when vi is the only text editor available, so it helps to at least know the basics.

On Mac OS X (and Linux), vi is symlinked to vim (vi improved), a more modern free software version. Vim It is the default editor when changing a crontab.

If you gave vi a whirl and don't see the beauty of it, give the nano editor a try. It also ships with Mac OS X.

note: a chunk of this small guide came from a web page I found long ago, but I don't remember where so I can't give proper credit. I've added and changed things from the original text.

 Vi has two modes, command and insert (really, three if you count replace mode). Command mode is used to navigate, search, and issue other commands. Insert mode is used to enter text. 
  Vi starts in command mode. 
 You can precede most commands with a number indicating how many times to perform a command. For example, entering 99 followed by the down arrow will move the cursor down 99 lines. "99x" will delete 99 characters. 
 While in command mode (case sensitive)
move the cursor with arrow keys; if there aren't any arrow keys, use j,k,h,l
i - change to insert mode (before cursor)
a - change to insert mode (after cursor)
A - change to insert mode (at end of line)
r - replace one character
R - overwrite text
x - delete one character
dd - delete one line
yy - yank line (copy)
p - paste deleted or yanked text after cursor
P - paste deleted or yanked text before cursor
G - go to end of the file
1G - go to top of the file
J - merge next line with this one
/ - search, follow / with text to find
:wq - write file and quit
:q! - quit without saving
%s/old/new/g - substitute; replace "old" with "new" on all lines
:g/pattern/d - delete all lines that match the pattern
  While in insert mode
ESC - change to command mode
any text typed is entered at the cursor
  Typical vi session Type "vi file.txt" at command prompt
Move cursor to where new text will be added
Type "i" to change to insert mode
Type new text
Type ESC to go back to command mode
type ":wq" and ENTER to write the file and quit



Sudo vs. Root: The Real Story
2008-12-11T10:02:00.000-08:00
In Mac OS X, the root account is disabled by default. The first user account created is added to the admin group and that user can use the sudo command to execute other commands as root. The conventional wisdom is that sudo is the most secure way to run root commands, but a closer look reveals a picture that is not so clear.

note: originally published March 21, 2006 on my old site, linuxboxadmin.com

What you get with sudo
 What are you really gaining by using sudo in the default Mac OS X configuration? First, you gain some comfort that nobody can login as root, either locally or remotely via SSH or FTP and tamper with your machine. Second, you get a log entry in /var/log/system.log every time sudo is used showing you who used it and what command was executed. These appear good enough reasons to endure the slight inconvenience of using sudo. 
 However, the way sudo is configured out of the box, you only need to enter your own password for authentication. This means that if someone guesses your password or steals it (and has access to it locally or via SSH), they can take over your box just as if you had root enabled. 
   Worse, if you execute sudo -s to start a root shell, the only thing that shows up in your system.log is this:

Mar 20 07:49:12 my-mac-mini sudo:   username : TTY=ttyp3 ; PWD=/Users/
username ; USER=root ; COMMAND=/bin/bash

Every other command after starting a root shell does NOT get logged at all. All you can tell from this is when someone started the root shell. Whatever happened after that is a mystery. The same problem exists if a command is executed that permits shell escapes like many text editors, telnet programs, etc. So, in fact, using sudo has gained us absolutely nothing over enabling and using root. 
   These deficiencies can be mitigated, and we'll get to that later. 
  Securing the root account
If you enable the root account, there are a couple of precautions you should take. First, give root a different password than your user account. 
  You can prevent root logins to SSH by changing this line in the sshd configuration file, /private/etc/sshd_config:

#PermitRootLogin yes

to this:

PermitRootLogin no 
  To go one step further, disable all password logins to SSH and allow only public key authentication. This is how I configure my Linux servers. There are many fine resources on the web that describe the gory details of using SSH public key authentication. 
  FTP logins by root are disabled by default since the root account is listed in the /etc/ftpusers file. Users listed in that file are not allowed to login using FTP. 
  Finally, disable user access to sudo by commenting out the %admin line in /private/etc/sudoers:

#%admin  ALL=(ALL) ALL
 With two minor configuration changes, we have a system that is arguably more secure than the default system using sudo. Why? Because if someone guesses or steals your user password, they can't use sudo to take over the machine. They still have to guess the root password. Of course, if they have a local account, they may be able to use a privilege escalation vulnerability to gain root access, but that is an issue for Apple. 
  Back to sudo
Is there a way to make the sudo configuration more secure? There are many things that can be done to improve the default settings. Here are a couple. 
 The most obvious change is to require a different password than your user password to authenticate. This can be done while keeping root logins disabled with a little trickery. First, enable the root account, change the root password, then use Netinfo Manager to change the root shell to /usr/bin/false. Any attempt to login as root will immediately end. Then, you can force sudo to require the root password by adding this line to /private/etc/sudoers:

Defaults:ALL rootpw
  Another security enhancement is to set up restrictions by user, and listing specific commands that are allowed to be run using sudo. By limiting the commands that can be run, you can limit the damage that can be done by a user account. This means changing the line in /private/etc/sudoers that grants all commands to users in the admin group. Check the sudoers man page for the details. 
 With these changes in place, sudo becomes much more secure, and is probably safer than using root directly. You should still change the SSH configuration to deny root logins and use public key authentication. 
  The real story
I've made arguments and suggestions for using the root account and for using sudo. But consideration should be given to the role of the computer and primary user(s) before making a decision on which may work best for you. 
 The main goal of sudo is to allow users limited access to root commands for the purpose of distributing the sysadmin load. On a single user box, you are only distributing the load to yourself. If you take a few precautions, enabling the root user is perfectly acceptable and can be more secure than the default configuration using sudo. On a multi-user box, sudo adds value and may be the best way to go. Given its limitations, the notion that sudo is always the best choice is dubious. The real story is it depends on the configuration. 



Find suid/sgid files
2008-12-11T09:57:00.000-08:00
There are special permission that files in Unix-like systems can have called "set user id" (suid) and "set group id" (sgid). When set, these permissions alter how the programs are run. When suid is set, the program runs as the user who owns the file. When sgid is set, it runs as the group that owns the file. While these permissions are very useful, they can also be dangerous. You especially want to limit the number of programs that run as user root since they can write to any part of the system. This tip shows how to use the find program to locate all suid/sgid files. You need to run these commands as root in order to search the entire system.

Find all SUID root files:
find / -user root -perm -4000 -print

Find all SGID root files:
find / -group root -perm -2000 -print

Find all SUID and SGID files owned by anyone:
find / -perm -4000 -o -perm -2000 -print

Find all files that are not owned by any user:
find / -nouser -print

Find all files that are not owned by any group:
find / -nogroup -print

Find all symlinks and what they point to:
find / -type l -ls


File Timestamps
2008-12-11T09:55:00.000-08:00
Each file has three timestamps associated with it (stored as the number of seconds since the Epoch, Jan 1, 1970). The three timestamps are: Access time (atime) - the last time the file was read
Modify time (mtime) - the last time the file contents were changed
Change time (ctime) - the last time the file permissions were changed
 

 In a long directory listing, the timestamp shown is the Modify time (mtime). To see all timestamps and a lot of other useful information, use the stat program with the verbose option (-x):

  stat -x filename 
  Here is sample output from stat:

keithw$ stat -x "Mona Lisa Overdrive.mp3"
  File: "Mona Lisa Overdrive.mp3"
  Size: 6853358      FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/  keithw)  Gid: (  501/  keithw)
Device: 14,9   Inode: 10208    Links: 1
Access: Fri May 25 11:46:30 2007
Modify: Fri Dec  8 16:38:54 2006
Change: Fri Dec  8 16:38:54 2006


BASH Basics
2008-12-11T09:54:00.001-08:00
Following are some basics on how BASH gets initialized and core programming constructs.

BASH initialization When you first login, bash reads these initialization files in order (if they exist):
 /etc/profile -- systemwide profile applies to all users
  Then, it looks for these files and executes the FIRST one it  finds (note that the file names start with a period):
$HOME/.bash_profile
$HOME/.bash_login
$HOME/.profile
 For interactive non-login shells, it executes:
$HOME/.bashrc 
 At logout, it executes:
$HOME/.bash_logout 
  Built-in shell variables:  These can be referenced interactively, but usually they are used in a shell script. A shell script is a plain text file that contains BASH commands executed in the order they appear. It is a program executed by BASH.
        $#      number of command line arguments    
        $?      exit value of last command    
        $$      process ID of current process    
        $!      process ID of last background process    
        $0      command name of the current process    
        $n      (n=1-9) the 1st thru 9th command line arguments    
        $*      all command line arguments    
        $@      all command line arguments, individually quoted ($1 $2 ...)    
 
  
If statement  if condition ; then
  commands
elif condition ; then
  commands
else
  commands
fi 
  Test the return status of the previous command:  if [ $? == 0 ] ; then
  commands
fi 
  Loops  while condition; do
  commands
done 
  for var in list; do
  commands
done 
  for (( expr1; expr2; expr3 )); do
  commands
done 
  Case statements The case statement can be used in place of a complex if statement.

case expression in
pattern)
  commands
   ;;
pattern)
  commands
   ;;
*
  commands
esac 



Deleting files with bad names
2008-12-11T09:50:00.000-08:00
If a file with a bad name gets accidentally created, such as a name that begins with a hyphen "-", it can't be deleted with a normal remove command (rm). Use the "--" option to tell rm that no more options follow, then it can delete the file.

To delete a file whose file name begins with "-":  rm -- -bad-file-name
Or
rm ./-bad-file-name 
 To delete a file with non-printable characters in the name:  Use shell wildcards, "?" for one character and "*" for zero or more characters. For example, if the file name "bad file name" can't be deleted, one of the spaces may in fact contain a hexademical value. Try:

rm bad?file?name

caution: use ls bad?file?name first to make sure you are not matching more files than you think with wildcards before deleting them. 



Mac/Linux/Windows file name friction
2008-12-11T09:43:00.000-08:00
In 1993, Microsoft added long file name support to Windows NT 3.1, allowing more descriptive names than the limited 8.3 DOS format. Mac users scoffed, having had long file names for nearly a decade, and because Windows still stored a DOS file name in the background. Linux was born with long file name a couple of years before it showed up in Windows. Today, long file names are well supported by all three operating systems though key differences remain.

Linux is the most sensitive

One of first culture shocks for people moving from Windows to Linux is the case sensitivity of file names. These files: "filename", "Filename", and "FileName" are the same file in Windows but three unique files in Linux.

The Mac OS X HFS+ and Windows NTFS file systems are case preserving, but not case sensitive. This means they will store and keep track of the case in a file name, but will ignore case when comparing file names. So, you can't have "filename" and "Filename" in the same directory. (note: an optional file system called HFSX, available in OS X 10.3+, is case sensitive.)

Linux and OS X files have more character(s)

Linux file names can be up to 255 characters long and can be made up of any characters except forward slash (/) and NUL. Using special characters like those recognized by the shell takes extra effort. You can trick the shell by either enclosing the name in single quotes ('icky?filename') or escaping it with a backslash (icky?filename). You can also use non-printable characters in a file name, but I can't think of a good reason to do so. To avoid confusion, most people stick to alphanumerics, periods, spaces, underscores, and hyphens.

OS X supports up to 255 characters and can use the same characters as Linux, except for a colon (:). However, the Finder may have trouble with bizarre file names that can be created in the shell.

Windows file names can be up to 255 characters, but that includes the full path. A lot characters are wasted if the default storage location is used: "C:\Documents and Settings\USER\My Documents". Windows does not allow names to contain any of these characters: * |  / :  ?

Avoid using these characaters
  for maximum portability        
        Asterisk      *    
        Colon      :    
        Back slash
     \
   
        Forward slash      /    
        Less Than      <    
        Greater Than      >    
        Pipe      |    
        Quote      "    
        Question mark      ?    
        all non-printable characters

Portability 
Even though you can use odd characters in a Linux or OS X file name, it's not a great idea. Using sensible names allows a smooth exchange of files with other operating systems where an odd character may be invalid. For maximum portability, avoid using characters that are illegal in any of the operating systems. 
 The lowercase-hyphen rule My preference is to always use lowercase letters and hyphens to replace spaces. For example, "this-is-a-long-filename.txt", instead of "This is a long filename.txt". 
 I'm not sure where or when I picked up the lowercase habit but it has been with me for some time. Since I spend a lot of quality time with web files, it relieves me of thinking about the case of file names. I like hyphens over underscores because underscores don't always show up clearly in anchor tag hyperlinks. Another web bias. Even though it is not required in Linux or OS X, I add a file type extension (three or four characters) mostly for my own benefit. I use the same naming convention regardless of operating system. 
 There are two exceptions I make to the lowercase-hyphen rule. The first is audio files, since ripping software tends to default file names to the track title with mixed case and spaces. The other exception is when I exchange documents with someone who has already named a file. The tab completion feature of BASH takes the edge off working with whatever is thrown at me. 
 Taming file names Modern operating systems provide a lot of flexibility in naming files. I probably expend too many brain cycles thinking about file names, but I suspect most people have a preference.




Boot key sequences
2008-12-11T08:57:00.000-08:00
Following are some of the most commonly used boot key sequences for recent Macs (with Open Firmware or Extended Firmware Interface).
c = boot from CD
n = attempt to boot from network server (using BOOTP or TFTP)
t = boot into FireWire Target Disk mode
shift = safe mode, disable login items and non-essential kernel extensions (OS X, 10.1.3 and later)
hold down mouse = eject CD/DVD from drive
cmd-v = verbose mode, show console messages during boot
cmd-s = boot into single user mode
cmd-opt-shift-delete = boot from external disk drive (or CD)
opt = show Open Firmware system picker (PowerPC only)
cmd-opt-o-f = boot into Open Firmware (PowerPC only)
cmd-opt-p-r = reset parameter RAM (PRAM)

$#	number of command line arguments
$?	exit value of last command
$$	process ID of current process
$!	process ID of last background process
$0	command name of the current process
$n	(n=1-9) the 1st thru 9th command line arguments
$*	all command line arguments
$@	all command line arguments, individually quoted ($1 $2 ...)

Avoid using these characaters for maximum portability
Asterisk	*
Colon	:
Back slash	\
Forward slash	/
Less Than	<
Greater Than	>
Pipe	\|
Quote	"
Question mark	?
all non-printable characters