Apple System Logger

Despite appearances, the system logging facility is unique on OS X 10.4 and newer. The syslogd daemon used on most Unix-like operating systems was replaced with the Apple System Logger (ASL).

Instead of using a new program name, Apple just used the traditional Unix system logger name. I think this can be a source of confusion and was not a great decision.

Binary Logs, then Text Logs

Instead of logging everything to text log files in /var/log, ASL logs all messages to a binary database file, /var/log/asl.db. From there, certain messages are written to the more familiar text based logs based on settings in /etc/syslog.conf.

In a way, Apple hacked the syslogd process, but did so in a way that maintains some level of backward compatibility with the knowledge and tools of system administrators coming from other Unix-like systems. I admit I was quite surprised to find out how this mechanism has been implemented by Apple.

The binary asl.db database is cleaned periodically, removing older messages from the database and keeping it from growing indefinately. The text based logs are still compressed and rotated as usual.

Reading log files

You can use the GUI utility, Console.app, to browse the Apple binary log database. It has a nice case insensitive search ability.

The command line interface to the Apple binary log is syslog.

Entering syslog with no parameters dumps the entire database. You can pipe the output through a grep command looking for a text string. For example,

syslog | grep Error

returned a large number of errors showing someone trying to login to my system via SSH as root:

Sat Dec 20 16:18:42 white com.apple.SecurityServer[22] : checkpw() returned -2; failed to authenticate user root (uid 0).

Yikes! Not to worry, when remote logins (SSH) are enabled, the default configuration does not allow root logins. According to the man page, you can also search for particular keys in the database using the -k option. Unfortunately, the list of valid keys is not in the man page.

Logging from a script

Finally, you can send a message to the log file using the -s parameter. Messages in the traditional syslog system are categorized by severity, or log level:

Emergency (level 0)
Alert (level 1)
Critical (level 2)
Error (level 3)
Warning (level 4)
Notice (level 5)
Info (level 6)
Debug (level 7)

To send a Warning message, use:

syslog -s -l 4 "this is a warning"

produces this in the log:

Sat Dec 20 18:42:29 white syslog[40323] : this is a warning


Disabling hardware via kernel extensions

You can disable hardware components in OS X by deleting (or renaming) the kernel extension directory that contains the driver for that piece of hardware.

Kernel extensions are stored in subdirectories in /System/Library/Extensions.

In some cases, there is more than one driver involved, so you need to delete or rename multiple directories.

Drivers for AirPort wireless:

• AppleAirPort.kext
• AppleAirPort2.kext
• AppleAirPortFW.kext

Drivers for Bluetooth wireless:

• IOBluetoothFamily.kext
• IOBluetoothHIDDriver.kext

Drivers for external mass storage (external hard disks or USB keys):

• IOUSBMassStorageClass.kext
• IOFireWireSerialBusProtocolTransport.kext

You may have delete or rename the directories again after a system update because an update may restore the drivers. After removing the drivers, reboot to make sure the hardware is disabled.

Another way to enable and disable root

I stumbled across another way to enable and disable root. There is a program called dsenableroot that can be used to both enable and disable the root account.

The full path is /usr/sbin/dsenableroot and you must be an admin level user to run it. In both cases, you will be prompted for your own user password. When root is enabled, you are prompted to set the root password.

This sequence enables root:

dsenableroot
username = keithw
user password:
root password:
verify root password:

dsenableroot:: ***Successfully enabled root user.


This sequence disables root:

dsenableroot -d
username = keithw
user password:

dsenableroot:: ***Successfully disabled root user.


I generally like to have root enabled. Here is my older post on enabling root.

Power settings from the command line

By request, I did a little digging to find out how to view and set power settings from the command line.

The command to use is pmset.

Instead of inventing the wheel again, please see this most excellent article at University of Utah.

Speeding MacBook 802.11g connections

My wife got a new MacBook a few weeks ago and was complaining of slow 802.11g wireless speeds at home. Normal troubleshooting went nowhere. Configuring the access point a little differently improved the situation.

My wife took the MacBook into an Apple store and diagnostics were run on it showing no connection or performance issues. At home, it was still slow. This was odd because I have a slightly older MacBook and an even older Linux laptop working fine on the same wireless network.

I searched the Interwebs and found a post on an Apple related forum, forget where now, that suggested Macs liked Channel 11 on 802.11g networks.

With nothing to lose, I logged into the LinkSys access point and changed it to listen on Channel 11 (2.462 Ghz) by default. After this change, the new MacBook wireless performance was much better. I have no idea if there is some affinity for Channel 11 in Apple's hardware or software stack, or if there is less interference on that frequency, or if there is some other explanation. In any case, I thought it was worth sharing.

Intro to ipfw

The low level firewall built into FreeBSD, ipfw, is also a part of OS X. It is one of the juicy UNIX bits that came along for the ride. The program itself lives at /sbin/ipfw and is a stateful packet filter capable of controlling access to and from specific IP addresses, ports, or combinations of each.

The Firewall in System Preferences

The firewall settings in Preferences are application level, and are separate from the low level rules set by ipfw.

On Leopard, you can get to the Firewall settings by going into System Preferences, select the Security icon, then the Firewall tab. The default setting is to allow all incoming connections. This is a little misleading because OS X comes with network services generally locked down until you enable them.

If you enable sharing, for example web sharing via the Apache 2 server in the Sharing Preferences, you will see Web Sharing appear in the firewall box in System Preferences. For most people, this level of control is fine, especially if you have another firewall between you and the Internet. For advanced configurations, ipfw can meet your needs.

Listing ipfw rules

To list all active ipfw rules:
sudo ipfw list

Each rule is assigned a number up to 65535. Lower numbered rules are executed first, and if they don't apply to the packet in question, it moves down the list until it matches a rule. It is possible for a packet to be injected back into the rules, but usually a packet matches one rule and is either accepted, denied, or routed to a new destination.

Clearing all ipfw rules

To list all active ipfw rules:
sudo ipfw flush

Creating custom rules

The command line interface to ipfw is somewhat arcane, and you can get yourself into trouble if you don't know exactly what ports you need open and why. Here is an article that has some sample configurations and command line syntax examples.

While I normally prefer the command line to GUI tools, if you are just starting to use ipfw, I recommend the very friendly NoobProof, a program that offers a useful GUI front end to ipfw.

NoobProof comes preconfigured with all the common Mac services that you can enable or disable at the network level using ipfw. It also has an install tool to create a script that saves your configuration and starts it automatically after a reboot. It is a truly wonderful tool and a much safer way to get started with ipfw.

Airport command line utility

OS X Daily reports on a hidden but useful command line utility to monitor and tweak wireless connections called airport.

Date validation in Ruby using the Date object

Ruby has a date library to help you manage and manipulate dates. The date library provides Date and Date/Time objects that can be used to validate dates. In addition to straight Ruby, this helps work around a particularly nasty problem in version 1.x scaffolding date fields created by Rails.

The Date object

To use the date library, add this line near the top of your Ruby program:

require 'date'

The date library provide the Date object and the DateTime object, which adds time attributes to the Date object. To create a new Date object, pass the year, month, and day to the new Date constructor:

mydate = Date.new(2008, 7, 11)

To get the current date, use the Date.now method. You can access different attributes of a date object like this:

mydate.year
mydate.month
mydate.day

Converting a date object to a string returns a "YYYY-MM-DD" format. For example:

mydate.to_s
"2008-07-11"


A simple validate function

One way to test for a valid date is to try to create a Date object. If the object is created, the date is valid, and if not, the date is invalid. Here is a function that accepts year, month, and day, then returns true if the date is valid and false if the date is invalid.

  def test_date(yyyy, mm, dd)
   begin
     @mydate = Date.new(yyyy, mm, dd)
     return true
   rescue ArgumentError
     return false
   end
 end

The creation of the date object is wrapped in a begin...rescue...end block so that the error can be trapped if the date is invalid. Ruby throws an ArgumentError if the date is invalid and in that case, the function returns false. This version uses an instance variable (@mydate) because I wrote it to be used in a Rails application.

The nasty date problem in 1.x Rails scaffolding

In version 1.x of Rails scaffolding, date fields in the edit view are generated with three drop down boxes, one for the year, one for the month, and one for the day. However, you are free to select invalid month/day combinations like February 30 and November 31. When these parameters are passed to the Rails update function in a scaffolding generated controller, it fails.

The crude solution I used to work around it was to edit the date prior to the attempt to save the record. In this case, the view sends the year to the controller in the dob(1i) field, the month in dob(2i), and the day in dob(3i). I convert them to integers, then call the function above to validate the date. If the date is invalid, I set a flash message and return to the edit screen.

    # date validation code
   @allparms = params[:dependent]
   @yyyy = @allparms["dob(1i)"].to_i
   @mm = @allparms["dob(2i)"].to_i
   @dd = @allparms["dob(3i)"].to_i

   if test_date(@yyyy, @mm, @dd) == false
     flash[:error] = "Invalid date " + @mm.to_s + "/" + @dd.to_s + "/" + @yyyy.to_s
     redirect_to :action => 'edit', :id => params[:id]
     return
   end

It seems like this rather basic problem would have been worked out in the Rails scaffolding, but it wasn't. I haven't tested Rails 2.0 scaffolding to see if it has similar issues. Other people have come up with more comprehensive solutions, like the Rails date kit. Take a look at something like that if you want more than a quick and dirty solution.

Beginning a Rails app -- for beginners

After some trial and error, I got my first significant Ruby on Rails application off the ground. By significant, I mean a dozen related tables, about 20 controllers, and some tricky business logic. I had been exploring Rails for a while and created a few toy applications, but hammering this one out still presented a challenge.

If you are used to programming CGI, PHP, or any application not built on a Model-View-Controller (MVC) framework, the learning curve is steep. While I can't offer up best practices -- I still have too much to learn -- I can offer up how-the-hell-do-I-get-started practices. For my own benefit, I created a list of steps to ease the pain of getting started.

Don't start here

As I mentioned, unless you are coming from another MVC framework, the learning curve is kind of steep. Rails has lots of nooks and crannies, lots of helpers and shortcuts that are not obvious at first. So, even though this is a beginning Rails article, it is not a good place to start learning about Rails. I suggest starting with a good book.

The three books I always have at hand when working in Rails are:

• Programming Ruby: The Pragmatic Programmers' Guide (pickaxe)
• Agile Web Development with Rails
• Rails Cookbook

Also, I spent no time telling you how to install or set up Rails on your system. There are many places to find such information, and if you are running a Mac, you already have Rails installed, though you may need to set up your database.

Four terminals at Rails central

I do my text editing with vim (and sometimes nano), performing all work in the Terminal. Textmate and other GUI editors are popular but I prefer to stay in the Terminal. I have found that 4 terminal windows (or tabs) is ideal for working on a Rails application. That's 4 terminals for each Rails application, so if I was going to work on two at the same time, (unlikely), I would open 8.

I keep the Terminals ordered left to right and dedicate the first to working on Models, the next one for Views, the next one for Controllers, and the fourth one for performing other tasks such as debugging, reviewing logs, running scripts, etc. Keeping them in MVC order helps me stay organized while working and lets me switch between related views and controllers quickly.

Of course, I also have a browser window open to test changes as I go.

Start here

Keep in mind, these are not rules, just basic guidelines that can help you approach a new project.

1. create the app structure with "rails appname", this command generates the directory structure for the application and all the configuration files
2. create the database and tables (outside of rails), this is not the "rails" way, using migrations, but I prefer to do create/modify my database outside of rails
3. configure /config/database.yml with database details
4. create models, one per database table in app/models/
5. add validate helpers and validate methods to models in app/models/
6. create initial controllers/views using scaffolding with "ruby script/generate scaffold Model-name"; run once for each table/model using the singular form of the table name for the controller. (The model name is capitalized and singular).
7. add business logic to a controller and set instance variables with data that will be needed in the view
8. customize the default template in /app/views/layouts/. Layouts are used to hold common HTML and embedded ruby such as headers and footers for your pages. Usually, there is one layout per controller.
9. fill in the views (often rhtml files), one for each method in the controller. Views and layouts control the presentation.
10. repeat each step above starting with step 6 (the scaffolding step) for each table/model in your application, you can come back and add or change models later
11. create new controllers not related to any models with "ruby script/generate controller Controller-name" and add methods and views as needed
12. edit config/routes.rb to set the default URL for the application, rename public/index.html to index.html.old,
    e.g. map.connect '', :controller => "main", :action => "list"
    it must be the first map in the routes file.

For pure data entry forms, create an empty controller method with a corresponding view. The view will contain the form and call a different method to accept the data and display the results (in yet another view).

If authentication is needed, use a "before_filter" at the top of each controller to handle logins and set session data.

In Rails version prior to 2.0, the scaffold script generates non-RESTful code, while in 2.0 and later, it generates RESTful code. This is a concern if you were used to the way Rails worked before 2.0. I am not sure RESTful is really better, but either way it can be confusing if you upgrade to Rails 2.0 in the middle of developing an application and find that scaffolding no longer works the same way.

Rails is flexible

One of nice things about Rails is that it is flexible. You can use Rails helpers when constructing your views and web forms, or you can code them using standard HTML. If I was not comfortable using a particular form helper or could not get the syntax just right, I fell back to coding it in regular HTML. Rails had no problems merging form helpers and regular HTML in the same form within a view. In many parts of Rails, it is not all or nothing. You can use all, some, or none of the shortcuts.

The ease of modifying applications is another strength of Rails. It is relatively easy to add new functionality, tables, or features. You don't have to figure out everything before you start. It is also easy to rewire your application if you don't like the way it flows between screens or the way the pages are linked together.

Testing

I think the official best practice to test a Rails app is to write test scripts that assert conditions and report problems found. I haven't used test scripts yet, my skills are not advanced enough. Up to now, I have relied on three other techniques.

The first is using the "script/debugger" to stop my application at certain points so I can display the values of objects and variables. I think the script/debugger was either removed or replaced in Rails 2.0.

The quick and dirty way to test is to write data to :flash and view the contents of variables that way. The flash is a built in hash, part of the session data, used to display informational or error messages to the user. It is an easy way to debug simple problems.

For deeper problems, I write data to the log file using the logger object. It takes a little time to sift through all the stuff in the development log file, but you can see trends and what is passing through the application before and after the trouble spot.

Railsbrain

Finally, I want to point out an extremely useful reference web site for working with Rails: Railsbrain. It has an Ajaxy API reference for multiple versions of Rails along with syntax and sample code.

I expect to come back to this article in six months and laugh at the inefficiency of the way I started doing Rails, but for now, it was enough to get that first useful application done.

Ruby Snippets

As I get deeper into the Ruby world, it is helpful to have a few snippets of working code and syntax for common tasks to copy as needed. I expect to add to this tiny reference over time.

updated July 9, 2008 to add a one liner to remove DOS line endings from a text file.

Formatting numbers

To format numbers, use the sprintf method.

x = 1.00
puts "x is $" + sprintf("%#0.2f", x)
puts "x is " + sprintf("%#2d", x)

The output is:

x is $1.00
x is  1

The defined? method

The defined? method can be used on any object to see if it exists.

x = 1
if defined? x
 puts "x is defined"
end

Case statements

Case statements are a more elegant way to handle a large number of conditions than using nested if statements.

x = 3
case x
when "1"
 result = "x is 1"
when "2"
 result = "x is 2"
when "3"
 result = "x is 3"
else
 result = "x is something else"
end

Substrings

Ruby has (at least) two ways to reference parts of a string object. One is to use the slice method:

x = "abcde"
y = x.slice(0,1)

The result is y = "a".

x = "abcde"
y = x.slice(-1,1)

The result is y = "e".

x = "abcde"
y = x.slice(2,3)

The result is y = "cde".

The second way is to treat the string object as an array of characters like the C programming language.

x = "abcde"
y = x[2,3]

The result is y = "cde".

One liner to edit a file in place

These one liners are also in the general Ruby article.

This command edits a file in place, performing a global text search/replace. The -p switch tells ruby to place your code in the loop while gets; ...; print; end. The -i tells ruby to edit files in place, and -e indicates that a one line program follows.

ruby -p -i -e '$_.gsub!(/248/,"949")' file.txt

One liner to remove DOS line endings from a file

A Unix or Mac text file uses a single line feed character to mark the end of a line (hex 0A). Files created on DOS or Windows use two characters, carriage return and line feed (hex 0D 0A). This command edits a file in place, removing the carriage returns.

ruby -p -i -e '$_.gsub!(/\x0D/,"")' file.txt

One liner to edit files in place and backup each one to *.old

ruby -p -i.old -e '$_.gsub!(/248/,"949")' *.txt

Boot Camp

Boot Camp is a set of tools to make dual booting an Intel Mac with Windows XP or Vista easy. The main reason you might want to dual boot as opposed to running Windows in emulation is for the extra performance from running natively. You might need this for gaming or a demanding application like CAD or photoshop where running in a virtual is just too slow.

You must have a CD or DVD with the full version of Windows XP or Vista. An upgrade disk will not work.

Update firmware

Before beginning the process, Apple recommends updating the firmware on your Mac to the latest version. If you are applying all software updates, then you are probably already at the latest revision. If you are not sure, run the Software Update command from the Apple menu.

Boot Camp Assistant

The next step is to run the Boot Camp Assistant program in the Utilities application folder. The first button in the assistant prompts you to print the installation and setup guide. I highly recommend you take that advice. I didn't and (temporarily) ended up with a Mac that would not boot. The guide is 25 pages long and covers all the details to get Windows installed. For that reason, I'll just cover the juicy bits and tell you the mistakes I made along the way.

The assistant provides a disk partitioning tool that lets you carve out part of your disk for Windows. I allocated 12 GB and proceeded to installation.

Installing Windows

I don't own Vista, so the rest of this article applies to Windows XP Pro SP2.

The installation of Windows works exactly like it does on any PC. When you get to the part where you have to choose what partition to install Windows on, select the one labeled [BOOTCAMP]. Also, it is important to have Windows format the partition (as either NTFS or FAT32), or it will not be able to boot back into the partition after copying its files to disk. The mistake I made was not formatting the disk. Even though the Windows installer could copy files to the disk, it was not able to boot back into the partition, leaving my Mac temporarily out of order.

I was able to get back into OS X by holding down the mouse button during boot, which ejected the Windows CD. Then I printed the installation guide, where I learned about the formatting requirement.

Installing Windows drivers

After Windows is installed, the next step is to install all the drivers for the Apple hardware. This was one of the best thought out parts of the process shows Apple's superior focus on human interface design. All you have to do is insert the OS X DVD while in Windows and it will start the driver installation wizard. Apple includes drivers for the video, audio, built-in camera, wireless airport, bluetooth, and more. It was the most painless Windows installation I have ever done.

Booting between OS X and Windows

To boot into OS X from Windows, you have two options. The first is to right click on the Boot Camp tray icon that gets installed during the driver procedure and select the option to boot back into OS X. You can also choose to make OS X the default boot option in the Windows Boot Camp control panel applet.

To boot into Windows from OS X, go to System Preferences / Start Up Disk and select the Windows partition as the default.

Finally, you can select which operating system to start at boot time by holding down the Option button.

Keyboard quirks on a Macbook

The differences in Apple and Windows keyboards is a source of minor annoyance. The keys are mapped in a reasonably intelligent way, but there are enough differences between the systems that some Windows features are awkard to use.

For example, to perform a right click, you hold two fingers on the trackpad and click the mouse button -- not a natural maneuver. The installation guide has a table that shows which keys and combinations map to Windows equivalents. Fortunately, most applications have some ability to map keys to make using them more comfortable.

Getting rid of Windows

Finally, the Boot Camp Assistant program has an option that lets you delete the Windows partition and restore the space set aside for use with OS X again. So, experimenting with a dual boot set up is not a one way ticket. Easy to try, easy to undo.

Ruby

Ruby is a popular text processing language that has gained a lot of momentum over the last few years, mostly due to the rise of Ruby on Rails. It has a lot of the good things from Perl, but implements a fully object oriented model and syntax.

Ruby is part of the default install of OS X.

Working with Ruby gems

Gems are Ruby packages that can be installed to extend the language or provide additional functions. Gems are managed using the gem command. To find all Ruby gems (packages) currently installed:

gem list

To install a new gem:

gem install package

To update the gems package manager:

update_rubygems

Taint mode

For untrusted scripts, add -T flag to ruby at the start of the script:

#!/usr/bin/ruby -T

One liner to edit a file in place

This command edits a file in place, performing a global text search/replace. The -p switch tells ruby to place your code in the loop while gets; ...; print; end. The -i tells ruby to edit files in place, and -e indicates that a one line program follows.

ruby -p -i -e '$_.gsub!(/248/,"949")' file.txt

One liner to remove DOS line endings from a file

A Unix or Mac text file uses a single line feed character to mark the end of a line (hex 0A). Files created on DOS or Windows use two characters, carriage return and line feed (hex 0D 0A). This command edits a file in place, removing the carriage returns.

ruby -p -i -e '$_.gsub!(/\x0D/,"")' file.txt

One liner to edit files in place and backup each one to *.old

ruby -p -i.old -e '$_.gsub!(/248/,"949")' *.txt

Intel vs. PowerPC Macs

I recently purchased a new Intel Macbook to replace my trusty iBook G4. Much has been written about the Apple's switch from PowerPC to Intel processors and there is no denying the market power of Intel, but I have some comments now that I have spent a little time with both kinds of Mac.

RISC vs. CISC

When comparing Intel and PowerPC micro processors, a common theme used to be the Intel complex instruction set (CISC) vs. the IBM reduced instruction set (RISC). This line of reasoning may have had some merit in the 1980s or early 1990s, but processor architecture has blurred, with each family incorporating the best designs of the other. I don't read much about this any more because it mostly doesn't apply.

Big endian vs. little endian

The endianness of a processor refers to the byte order of data in memory. Specifically, big endian processors, (traditionally the PowerPC line), stored the most significant bytes first, while little endian, (the Intel line), stored the significant bytes last. If you have ever debugged data or instructions in memory, big endian is the logical way you would expect to see things, a more natural or human way to read memory. Little endian data looks garbled to me.

I started my programming life long ago on an IBM mainframe (big endian) and wrote quite a few lines of assembly language. I actually enjoyed working with the mainframe register architecture and assembly was a joy. When I started working with Intel PCs, assembly language became a nightmare. The registers could be addressed partially using one mneumonic, fully with another, and the debugger showed bytes in the strange little endian order. It was a real buzz kill.

Today, with optimizing compilers and high level languages, it doesn't make much difference any more. Most modern languages are so high level (my current favorite is Ruby), that endianess is transparent. Some processors can even be programmed to change their endianess. So, other than personal preference, and a slight distrust of engineers that prefer little endian, this is a difference that doesn't carry much weight.

Power, heat, and batteries

PowerPC processors have nearly always been more power efficient than Intel. This translates into less heat (less time running a cooling fan), and longer battery life. The battery life between my new Intel Core 2 Duo Macbook and my G4 iBook is dramatic. Even though the G4 is over 2 years old, it still gets about an hour and half of extra battery life. Certainly, part of that is because it also has a smaller and dimmer screen, but the processor architecture is also a factor. I was shocked at how fast the Macbook battery ran out the first week I had it. It is still better than a Windows laptop, but a step backward compared to the G4.

Binaries

The operating system was not the only thing that had to be compiled to run on Intel processors. To take advantage of native speed, every application had to be recompiled. Some software vendors took their time building Intel binaries. Apple provided the "Universal binary" solution ahead of the switch so the application hit wasn't too harsh.

The virtual plus

Apart from the ability of Intel to supply Apple with enough processors, the switch enabled fast virtual machine technology that doesn't rely on software emulation to work. Applications like Parallels and Vmware Fusion give the Intel based Macs a vastly improved capability to run Windows, Linux, and other operating systems with OS X as the host. I tried running Linux on the free Q virtual machine on the G4 and it was almost too slow to use. Running VMware Fusion, I have a couple of Linux distributions running graphical X desktops at near native speed. In this sense, the switch to Intel was a virtual plus.

Does it matter?

How you weigh the positives and negatives above determines how you view the switch. In the end, the best things about a Mac still exist in the Intel world, and apart from the good folks at IBM, Motorola, and Intel, most people won't notice.

How to turn off Dashboard

The Dashboard is a handy feature of OS X to keep information flakes and small tools at the ready.

To turn Dashboard off, run the following command from the Terminal:

defaults write com.apple.dashboard mcx-disabled -boolean yes

To turn Dashboard on:

defaults write com.apple.dashboard mcx-disabled -boolean no

Because Dashboard is a component of the Dock, you need to restart the Dock after making either change:

killall -1 Dock

Advanced BASH scripting guide

User forbade pointed out this great guide from the Linux Documentation Project:

Advanced BASH scripting (pdf)

It is also available in other formats.

Amazon S3 from the command line

Amazon.com offers a variety of web services to make building highly scalable web applications easier. One of those services is called the Simple Storage Service (S3). It provides a way to store and retrieve files (up to 5 GB each) on Amazon's distributed servers. S3 has both a SOAP and REST API. You can use the S3 REST API with only the native command line utilities installed by default in OS X Leopard (and Linux).

Why use Amazon S3?

Amazon S3 is designed to be robust, fast, and inexpensive. It is ideal for serving small images or small files for web sites. It is also a good solution for off site backups (less than 5 GB) because the storage is redundant and inexpensive. For very large backup files, it is not the best solution because of the file size limitation (which can be worked around in some cases), and because the bandwidth is prohibitive.

How does S3 work?

While S3 is cheap, it is not free. To use it, you first need to sign up with Amazon Web Services and obtain an access key ID and secret key. The access keys are required to authenticate to Amazon and to use the API.

Conceptually, S3 stores files (objects) in buckets. First, you create a bucket, then you store objects in the bucket. The combination of the bucket and object ID must be unique. To S3, all objects are binary blobs. You can attach some meta data to an object when it is stored, but S3 doesn't verify that the meta data accurately describes the object.

With the REST API, standard HTTP transactions are used to GET, PUT, and DELETE objects in buckets. There is no incremental update option. To update an object, upload and replace it with the new version.

Using S3 from BASH

As part of a consulting project, I was asked to upload rotating backup files to S3. The backup files consisted of compressed source code and database dumps.

I started by looking at the sample code on the Amazon developer site. I downloaded some working PHP code that used the REST API and began hacking on it. However, I ran into a serious problem when I tested uploading one of the backups. The sample code was designed for small files that could be uploaded in a single HTTP transaction. It had a limitation of 1 MB, while my backup files were around 250 MB. I could have tried to retool the code to break apart the backup file and upload in chunks but that seemed daunting. Instead, I went back to the developer site and found a set of BASH scripts that use the curl and openssl utilities to handle the heavy lifting.

There are two scripts in the package provided by the developer "nescafe5", hmac and s3. The s3 script is the main script and it calls hmac to calculate the hashes needed to authenticate to S3. Then, it uses curl to upload, download, list, or delete the objects in a bucket. It can also create and destroy buckets.

To use the s3 script, you need to store your Amazon secret key in a text file and set two environment variables. The INSTALL file included with the package has all the details. The only tricky part I ran into, and from the comments on Amazon, other people ran into, is how to create the secret key text file.

If you open a text editor like vim or nano, copy in your secret key and save the file, the editors will add a new line character (hex 0A) to the end of the file. The script requires the file to be exactly 40 bytes, the length of the secret key, or it will complain and stop. To create the text file without a new line, use the echo command:

echo -n "secret-key" > secret-key-file.txt

The -n switch tells echo to not include a new line character and results in a text file of exactly 40 bytes. Once I got the key file created correctly, the s3 script started working, and I was able to upload, download, and list objects in S3.

Here is an example of a test script I used:

#!/bin/bash
# export required variables
export S3_ACCESS_KEY_ID="99XC79990C39996AR999"
export S3_SECRET_ACCESS_KEY="secret-key-file.txt"

# store a file
./s3 put bucketname objectname /path/to/local/file

# list objects in a bucket
./s3 ls bucketname

# download a file
./s3 get bucketname objectname /path/to/local/file

Mac and Linux friendly

I tested the s3 scripts on both OS X Leopard and Red Hat Enterprise Linux 4 and they worked perfectly on both. Amazon Web Services offers powerful solutions to some tough problems. Other services offered include the Elastic Compute Cloud (EC2) for virtual servers, Simple Queue Service (SQS) for passing messages between applications, and Mechanical Turk for delegating work to humans. There is some truly cutting edge stuff going on at Amazon.

Say say say

The say command tells your Mac to talk back, reciting either a string of text or reading an entire file.

To have the computer read a string of text

Open a Terminal and type:
say "This is a string of text."

To have the computer read text from a file

Open a Terminal and type:
say -f /path/to/text/file

Using a different voice

The speech synthesis system comes with several voices. You can see all the voices available in System Preferences / Speech. The default voice selected in System Preferences is used unless a different one is specified with the -v switch. For example, to use the "Kathy" voice:

say -v Kathy "This is a string of text."

Installing a .dmg application from the command line

An intrepid reader asked the following question:

How do you install a .dmg package from the command line?

Many applications are distributed as disk images, a compressed binary format. If you double click a disk image in the Finder, it is mounted automatically. Once mounted, installation of the application is typically done by dragging an icon to the Applications folder. The same can be accomplished from the command line using two commands, hdiutil and cp.

The following steps show the installation of a popular VNC client for OS X called "Chicken of the VNC". It can be used as a remote desktop client for Linux, Mac, or Windows hosts.

The download file is named "cotvnc-20b4.dmg". Here are the steps needed to install it remotely from the command line.
note: this technique can be used from a local Terminal window or a remote SSH connection.

Mount the disk image

The first step is to mount (or attach) the disk image. From the command line, use:
hdiutil mount cotvnc-20b4.dmg
I received the following output:

Checksumming Driver Descriptor Map (DDM : 0)…
    Driver Descriptor Map (DDM : 0): verified   CRC32 $767AD93D
Checksumming Apple (Apple_partition_map : 1)…
    Apple (Apple_partition_map : 1): verified   CRC32 $DD66DE0F
Checksumming disk image (Apple_HFS : 2)…
..............................................................................
         disk image (Apple_HFS : 2): verified   CRC32 $EF1F362F
Checksumming  (Apple_Free : 3)…
                   (Apple_Free : 3): verified   CRC32 $00000000
verified   CRC32 $F5A3FFA1
/dev/disk1           Apple_partition_scheme          
/dev/disk1s1         Apple_partition_map             
/dev/disk1s2         Apple_HFS                       /Volumes/Chicken of the VNC

A mounted disk image appears on the Desktop, in the Finder, and more importantly shows up as a directory in /Volumes. In this case, the last line of output from hdiutil showed exactly where the disk image was mounted.

Sometimes when a disk image is mounted, it will prompt you to agree to a license first. In that case, the text that would normally appear in a GUI dialog box instead appears in the Terminal window. Once you scroll to the bottom of the agreement, you can type in Y to continue or N to stop. The Firefox disk image is one example of a package that displays a license before mounting.

Install the application

Use the cp command to copy the application to /Applications:

sudo cp -R "/Volumes/Chicken of the VNC/Chicken of the VNC.app" /Applications

The -R switch means to copy recursively, in other words, copy everything from that location including all subdirectories and files below. It is important to leave off the trailing "/" from the "Chicken of the VNC.app" directory, or the command will not copy the directory itself, just the contents. After entering your password, the application will be installed and ready to use.

Most applications can simply be copied to the /Applications directory. However, some are distributed in a .pkg format and must be installed using the installer command instead of cp. To install a .pkg, use this command:

sudo installer -package /path/to/package -target "/Volumes/Macintosh HD"

Unmount the disk image

To tidy up, return to your home directory and unmount the disk image:

cd ~
hdiutil unmount "/Volumes/Chicken of the VNC/"

You should see this message after the unmount:

"/Volumes/Chicken of the VNC/" unmounted successfully.

Installing applications from a .dmg package at the command line is not something you need to do every day. But it is a nice tool to have if you want to install an application on a remote server or script the installation of a package to a group of desktop Macs.

Updating Apple software from the command line

Macs check for system updates on their own (by default) and prompt you when there are updates to apply to your system. If you want to update immediately, like after a fresh OS X install, you can check for updates by clicking on the Apple menu icon, then Software Update. However, if you want to update a system remotely -- a frequent need for remote servers -- you can do it from the command line.

The softwareupdate command

Use the softwareupdate (all one word) command to check for updates or install them. Here are some commonly used switches:

Switch	Effect
--list	list available updates
--install	install all available updates
--install packagename	install only packagename
--schedule on	turn automatic checking on
--schedule off	turn automatic checking off

If you check for new software and there is nothing to apply, you get output like this:

iBook-G4:~ keithw$ softwareupdate --list
Software Update Tool
Copyright 2002-2007 Apple

No new software available.

When updates are available, you get output like this:

iBook-G4:~ keithw$ softwareupdate --list
Software Update found the following new or updated software:
  * iLifeSupport82-8.2
       iLife Support (8.2), 2770K [recommended]
  * QuickTime-7.4.1
       QuickTime (7.4.1), 59940K [recommended] [restart]
  * MacOSXUpd10.5.2-10.5.2
       Mac OS X Update (10.5.2), 349324K [recommended] [restart]

To install updates, use the --install --all option:

iBook-G4:~ root# softwareupdate --install --all
Software Update Tool
Copyright 2002-2007 Apple

Downloading iLife Support       0..20..40..60..80..100
Verifying iLife Support
waiting iLife Support
waiting QuickTime
Downloading Mac OS X Update     0..20..40..60..80..100
Verifying Mac OS X Update
waiting Mac OS X Update
Installing iLife Support        0..20..40..60..80..100
Done iLife Support
Installing QuickTime    0..20..40..60..80..100
Done QuickTime
Installing Mac OS X Update      0..20..40..60..80..100
Done Mac OS X Update
Done.

You have installed one or more updates that requires that you restart your
computer.  Please restart immediately.

Syncing Palm devices with the Mac via Bluetooth

Until I figured out how to sync my Palm based Treo smartphone with the Palm Desktop on my iBook, I had no use for bluetooth. Before then, bluetooth was one of those technologies that was around but that I largely ignored. Now, the thought of syncing via USB cable seems so 20th century. As a bonus, I cover how to transfer camera images from the Treo to the Mac via bluetooth.

This article documents all the steps needed to set up wireless syncing via bluetooth between a Treo and Palm Desktop for the Mac. Since there are a variety of Palm based devices and OS X versions, it is offered as is. Just because it "works for me" doesn't mean it will work for you. At the least, you may get some useful hints for your situation.

Hardware

My main Mac is currently an iBook G4 1.33 Ghz running Leopard (OS X 10.5.1). My Palm device is a Treo 650 smartphone running Palm OS Garnet (5.4.8). Both devices have native bluetooth wireless hardware.

Bluetooth is a low power, short range radio technology (wireless) designed to link devices together in a personal area network. Bluetooth 1.x operates over distances of about 10 meters with a data transfer rate of 720 kbps. While the low data rate is useless for large files, it works great for syncing Palm data or sending a few images back and forth.

Installing Palm Desktop

My first stop was the Palm web site where I downloaded the latest Palm Desktop software for Mac. The version I downloaded was 4.2.1 rev D which came in disk image (.dmg) format. I mounted the disk image and ran the installer. At the end of the installation, a reboot is required, probably to load the background processes cleanly.

Bluetooth configuration on the Mac

After the reboot, I visited System Settings and went into the Sharing settings. There I enabled bluetooth sharing and required pairing for all actions. Pairing is the process of introducing bluetooth devices to one another using a shared passkey. I learned later that pairing was not really required for Palm syncing, but I prefer to use it for file transfer so I checked all the pairing boxes.

Configuring HotSync manager

Staying on the Mac, go to Applications and open the Palm folder containing all the Palm Desktop applications. Run the HotSync Manager, go to the Connection Settings tab, and enable the bluetooth-pda-sync-port.

The dialog box on my screen cut off the name, but if you select it, you can see the full name beneath the list of selections. It is the second choice in the list, below the bluetooth-modem.

As a final check of the Mac set up, I opened a Terminal and ran the following command to verify that the Palm Transport Monitor was running in the background:

iBook-G4:~ keithw$ ps aux | grep Palm
keithw 130 0.1 0.3 154888 3468 ?? S 6:01PM 0:04.04 /System/Library/Frameworks/Carbon.framework/Versions/A/Support/LaunchCFMApp /Applications/Palm/Palm Desktop/Contents/Resources/Palm Desktop Background.app/Contents/MacOSClassic/Palm Desktop Background
keithw 131 0.0 0.3 118108 3628 ?? S 6:01PM 0:00.79 /System/Library/Frameworks/Carbon.framework/Versions/A/Support/LaunchCFMApp /Applications/Palm/Transport Monitor/Contents/MacOSClassic/Transport Monitor

Bluetooth configuration on the Palm Treo

On the Treo, go to the Communications group and open the Bluetooth application (it uses the Bluetooth icon). Make sure bluetooth is "On" and Discoverable is set to "Yes".

Then, click the "Setup Devices" button and on the next screen, click the "HotSync Setup" button. This starts a wizard-like procedure with 4 steps. The instructions use PC terminology and may appear confusing, but all of the Mac side set up should be done. Click Next to continue.

Step 1: says to create a virtual serial port on your PC, just click Next.

Step 2: We've already enabled the bluetooth-pda-sync-port in HotSync Manager, so just click Next.

Step 3: The local serial port is already enabled on the Mac, so click Next.

Step 4: click on the Launch HotSync button, it should start the first HotSync immediately. The "HotSync Progress" dialog should appear on the Mac indicating the first HotSync is under way.

Depending on how much data you have on your Palm, the first HotSync may take a long time. It has to transfer all of your data over bluetooth to the Mac. After the first HotSync is complete, only changed data will be transfered and HotSyncs will typically be quick.

Transferring files via bluetooth

Like most cell phones today, the Treo has a built in camera. Even though it is low resolution, it is still useful and convenient. I was looking for a zero cost way to transfer photos from the Treo to the Mac. Bluetooth was the answer.

Bluetooth configuration on the Mac

See the section above with the same title for details and a screenshot for configuring bluetooth.

While pairing may not be absolutely required, I think it is the best practice to require it. Otherwise, anyone wandering by with a bluetooth device could upload files to your Mac. I can think of a lot of bad scenarios that might arise out of that configuration.

Pairing the Treo and Mac

On the Treo, open the Bluetooth application (it uses the Bluetooth icon). Make sure bluetooth is "On" and Discoverable is set to "Yes". Then, click the "Trusted Devices" button, then "Add Device".

Select the Mac (it must also be discoverable) and click OK.

The Treo will prompt for a passkey. Enter any sequence of digits (I only used 4 digits). Enter the same passkey on the Mac when prompted. Pairing is a one time set up procedure and you won't need to remember the passkey after they are paired.

Sending an image (or any file) to the Mac

To send a camera image file, open any image on the Treo in the Media application. Then, open the Media menu and select Send...

In the Send with dialog, choose Bluetooth..., then select the Mac you are paired with, then OK. The file will be sent and the Mac will prompt you to accept it.

After you accept it, the file is saved in the folder you configured in bluetooth sharing, the "Download" folder in my case. It will have a file name similar to "Photo_112307_003.jpg".

How to enable the OS X root user

The following information was found at spy-hill.com.

This is the easiest method to enable the "root" account on a Mac if you are more of a CLI person:

1. Log in on the Admin account. (Your normal, every day user account should not have administrative privileges!).

2. Open up a command shell in the Terminal application with
   Macintosh HD -> Applications -> Utilities -> Terminal.
   
   

   At the command prompt type this command:

   % sudo passwd root Enter Password: Changing password for root New password: Verify password:

   The first password you are asked for is the already existing password for the Admin account, to prove that you are authorized to make changes to this system. After that, you enter the new root password (twice, for verification). That is all, the "root" account is now enabled, with that password.

Screenshots from the command line

Use the screencapture utility to initiate a screenshot from the command line.

Take a screenshot and save the result to a file

This command can be used to take a screenshot with a 10 second delay (-T 10) and save the result in jpg format (-t jpg) to a file named screen.jpg:

screencapture -T 10 -t jpg screen.jpg

If the file format is not specified, a png file is saved. The time delay lets you change desktops if you are using Spaces (the multiple desktop feature in Leopard) before the screenshot is taken.

How to find out which files were installed with a Mac package

Installing a Mac package is usually a process of dragging an icon from an installer to the Applications folder. This creates a new directory in the Applications folder with the application files inside it. Sometimes, additional files are installed in other places. To see everything that was installed, use the /Library/Receipts directory and the lsbom utility.

An applications that follows Mac standards will leave a record of what was installed in the /Library/Receipts/app-name directory.

Within that directory, there should be a Contents subdirectory, and inside that, an Archive.bom file. The .bom file is a "bill of materials" with a list of all the files that belong to that application, the full path to their location, and other details like the UNIX permissions, file size, and checksum.

However, the bill of materials is a binary file. To extract information from it, use the list bill of materials command line utility, lsbom.

To list all files and related information in a BOM file:

lsbom BOMfile

To list only the files without related information in a BOM file:

lsbom -s BOMfile

Using hdiutil

The hdiutil program is a native Apple command line utility for working with disk images. It uses the DiskImages framework. Disk images (usually with a .dmg, .img, or .iso file name extension) are often used for distributing programs and for burning CDs/DVDs.

Creating a disk image from a folder

hdiutil create test.dmg -srcfolder /path/to/folder/

Once the disk image is created, it can be mounted, copied, or sent like any other file.

Mounting a disk image

To mount (or attach) a disk image, use:

hdiutil mount test.dmg

A mounted disk image appears on the Desktop, in the Finder, and shows up as a directory in /Volumes. All removable media, such as CDs, DVDs, external disks, and USB flash drives are mounted in /Volumes.

Unmounting a disk image

To unmount (or detach) a disk image, use:

hdiutil unmount /dev/device-name

The device name is usually something like /dev/disk3s2. You can also unmount it using the /Volumes/mountpoint if you know where it was mounted:

hdiutil unmount /Volumes/mountpoint

Burning an ISO to CD (or DVD)

First, load a blank CD, then:

hdiutil burn cd-image.iso

Create an encrypted disk image

This creates a 10 MB encrypted disk image and internally formats it as a journaled HFS+ file system (the OS X default):

hdiutil create -encryption -size 10m -volname encdata test.dmg -fs HFS+J

During the creation of the disk image, you will be prompted for a password that will allow access to the contents of the disk image. You must remember the password or anything you put into the image will be lost.

Apache Authentication and Authorization using LDAP

The Lightweight Directory Access Protocol (LDAP) is frequently used to implement a centralized directory server. There are two popular open source LDAP solutions, OpenLDAP and Red Hat Directory Server. According to the Apache documentation, Novell LDAP and iPlanet Directory Server are also supported. This article focuses on OpenLDAP, but the concepts and examples should be applicable to the others.

note: originally published October 31, 2007 on linux.com

Information in OpenLDAP

LDAP was designed as a simplified, or "lightweight", version of the ITU-T X.500 directory specification. The default set of schemas contain all of the information you would find in traditional Linux system files like /etc/passwd and /etc/group, or Sun's Network Information System (NIS). The schemas are malleable and are often extended to contain additional demographic information or customized for specific applications.

Following is an example of a typical LDAP user record in LDAP Data Interchange Format (LDIF):

dn: uid=keithw,ou=People,dc=company,dc=com
uid: keithw
cn: Keith Winston
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
userPassword: {crypt}$1$M/PZEwdp$KHjSay8JILX01YAHxjfc91
shadowLastChange: 13402
shadowMax: 99999
shadowWarning: 7
loginShell: /bin/bash
uidNumber: 2741
gidNumber: 420
homeDirectory: /home/keithw
gecos: Keith Winston

The LDAP database can be queried with a number of tools, including the command line ldapsearch program, one of the standard OpenLDAP utilities. If you are new to LDAP, the terminology and syntax may be difficult at first. Taking the time to learn the LDAP search syntax will pay off later if you want to craft advanced policies using non-standard attributes.

Configuring Apache 2.2

Apache modules have been available for LDAP since at least version 1.3. If you have used mod_auth_ldap in the past, you should be aware that the bundled authentication and authorization modules have been refactored in version 2.2. The latest LDAP modules are loaded with these directives, usually in the httpd.conf file:

LoadModule ldap_module /path/to/mod_ldap.so LoadModule authnz_ldap_module /path/to/mod_authnz_ldap.so

Once the modules are loaded, you can control access by querying the directory for particular attributes. The key directive to point Apache at the LDAP server is AuthLDAPUrl. A generic AuthLDAPUrl directive looks like this:

AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid

It defines the LDAP server, the base distinguished name (DN), the attribute to use in the search (usually Uid within the People organizational unit). For complex policies, you may need extra search filters.

The next few sections show working examples of directives to enforce common policies. Each set of directives can be placed in the main Apache configuration file or in .htaccess files.

Any valid user

This set of directives allows access to the current directory to all valid users in the LDAP directory. Apache will ask the browser for a user ID and password and check that against the directory. If you are familiar with Apache Basic Authentication, there are only a few new directives to learn.

Order deny,allow
Deny from All
AuthName "Company.com Intranet"
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid
Require valid-user
Satisfy any

AuthBasicProvider ldap is necessary so Apache knows to query an LDAP directory instead of a local file.

AuthzLDAPAuthoritative off must be explicitly set because the default setting is "on" and authentication attempts for valid-user will fail otherwise. This is a tricky setting because other policies, like Require ldap-user, need the setting to be "on". Setting this value off also allows other authentication methods to mixed with LDAP.

The Satisfy any directive is not strictly required in this case because we are only testing one condition.

List of users

This set of directives allows access to the current directory to the users listed in the Require ldap-user directive.

Order deny,allow
Deny from All
AuthName "Company.com Intranet"
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid
Require ldap-user keithw joeuser
Satisfy any

AuthzLDAPAuthoritative on could be omitted since the default setting is "on", but left here for clarity.

Note the AuthLDAPUrl setting does not change. As in previous examples, it searches the directory for a matching Uid.

Member of a group

This set of directives allows access to the current directory to users who are either primary or secondary members of the group specified in the Require ldap-group directive.

The group configuration may be the most difficult due to the schema design of directories that were converted from NIS (as mine was). Referring back to the user LDIF record, notice the gidNumber attribute has a value of 420, the number assigned to the infosys group in my directory. It corresponds to the primary group of the user. However, the LDAP entry for each group only lists users who are secondary members of the group, using the memberUid attribute. See below for a snippet of a group record:

dn: cn=infosys,ou=Group,dc=company,dc=com
objectClass: posixGroup
gidNumber: 420
memberUid: user1
memberUid: user2
memberUid: user3
...

We need another test, Require ldap-attribute, to pick up the primary users of the group because they are not listed with the group itself. Here are the Apache directives:

Order deny,allow
Deny from All
AuthName "Company.com Intranet"
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid
AuthLDAPGroupAttribute memberUid
AuthLDAPGroupAttributeIsDN off
Require ldap-group cn=infosys,ou=Group,dc=company,dc=com
Require ldap-attribute gidNumber=420
Satisfy any

AuthzLDAPAuthoritative on could be omitted since the default setting is "on", but left here for clarity.

AuthLDAPGroupAttribute memberUid indicates which attibute in the LDAP group record to match with the Uid. In this case, memberUid. A group record contains one memberUid attribute for each (non primary) member of the group.

AuthLDAPGroupAttributeIsDN off tells Apache to use the distinguished name of the client when checking for group membership. Otherwise, the username will be used. In my OpenLDAP directory, only the username was from NIS. The default setting is "on" so setting it off was required. An LDAP directory may store the entire distinguished name, so this setting may be change based on your directory.

Require ldap-group grants access to members of the "infosys" group. For multiple groups, add an additional directive for each.

Require ldap-attribute gidNumber=420 handles the primary users of group 420, the "infosys" group. Without this condition, primary users would be denied access. For multiple groups, add an additional directive for each.

The Satisfy any directive is required because we are testing multiple conditions and want the successful test of any condition to grant access.

Combination of users and groups

This example is a union of the user and group directives, but otherwise, there is nothing new.

Order deny,allow
Deny from All
AuthName "Company.com Intranet"
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthLDAPUrl ldap://ldap.company.com/ou=People,dc=company,dc=com?uid
AuthLDAPGroupAttribute memberUid
AuthLDAPGroupAttributeIsDN off
Require ldap-group cn=infosys,ou=Group,dc=company,dc=com
Require ldap-attribute gidNumber=420
Require ldap-user keithw joeuser
Satisfy any

Debug and deploy

Testing LDAP authentication from a web browser can be frustrating because the only thing you know is that access was granted or not. You don't get any kind of feedback on why something did not work. For verbose information on each step in the process, set the "LogLevel debug" option in Apache.

With debugging active, Apache will record the connection status to the LDAP server, what attributes and values were requested, what was returned, and why conditions were met or not met. It can be invaluable in fine tuning LDAP access controls.

Using diskutil

The diskutil program is a native Apple command line utility for manipulating disks, partitions, and RAID sets. This includes magnetic hard disks, CDs/DVDs, and flash drives. Most options, except "list", require root access.

Finding out about disks in your system

For information on all available disks and their partitioning, use:
diskutil list

For more detailed information on a particular disk or partition, use:
diskutil info disk-or-partition

The default Apple partitioning scheme uses the last physical partition on a disk for storing data. Here is sample output from diskutil list showing a hard disk and a CD. The UNIX device name is shown first, along with the contents of each partition:

/dev/disk0
  #:                       TYPE NAME                    SIZE       IDENTIFIER
  0:     Apple_partition_scheme                        *74.5 Gi    disk0
  1:        Apple_partition_map                         31.5 Ki    disk0s1
  2:                  Apple_HFS Macintosh HD            74.4 Gi    disk0s3
/dev/disk1
  #:                       TYPE NAME                    SIZE       IDENTIFIER
  0:        CD_partition_scheme                        *718.1 Mi   disk1
  1:     Apple_partition_scheme                         625.3 Mi   disk1s1
  2:        Apple_partition_map                         31.5 Ki    disk1s1s1
  3:                  Apple_HFS Dungeon Siege Disc 2    625.0 Mi   disk1s1s2

Here is sample output from diskutil info on a disk partition:

diskutil info /dev/disk0s3
  Device Identifier:        disk0s3
  Device Node:              /dev/disk0s3
  Part Of Whole:            disk0
  Device / Media Name:      Untitled 3

  Volume Name:              Macintosh HD
  Mount Point:              /
  File System:              Journaled HFS+
                            Journal size 8192 KB at offset 0x256000
  Owners:                   Enabled

  Partition Type:           Apple_HFS
  Bootable:                 Is bootable
  Media Type:               Generic
  Protocol:                 ATA
  SMART Status:             Verified
  Volume UUID:              EE8C4FFD-6C4B-302D-B096-DCF81D4E13FB

  Total Size:               74.4 Gi (79892103168 B) (156039264 512-byte blocks)
  Free Space:               26.8 Gi (28731383808 B) (56115984 512-byte blocks)

  Read Only:                No
  Ejectable:                No
  Whole:                    No
  Internal:                 Yes

Checking partitions for integrity and fixing them

You can use diskutil to check the file system data structure of a partition (e.g., /dev/disk0s3) with:
diskutil verifyVolume partition

If errors are you found, you can fix them with:
diskutil repairVolume partition

Checking partitions for UNIX permission problems and repairing them

You can use diskutil to check the UNIX permissions on a partition with:
diskutil verifyPermissions partition

If errors are you found, you can fix them with:
diskutil repairPermissions partition

If permissions get accidentally changed on some system files, it could cause strange behavior or disable certain features of the system.

Finding out about RAID sets

RAID is usually used in servers to provide additional protection from hard disk failure. For information on RAID sets, use:
diskutil listRAID

Other diskutil options

In addition to the options listed above, diskutil can be used to reformat disks or partitions, erase writable CDs/DVDs, securely erase data, etc. Here are some of the other features:

• u[n]mount - Unmount a single volume
• unmountDisk - Unmount an entire disk (all volumes)
• eject - Eject a removable disk
• mount - Mount a single volume
• mountDisk - Mount an entire disk (all mountable volumes)
• eraseDisk - Erase an existing disk, removing all volumes
• eraseVolume - Erase an existing volume
• reformat - Reformat an existing volume
• eraseOptical - Erase an optical media (CD/RW, DVD/RW, etc.)
• zeroDisk - Erase a disk, writing zeros to the media
• randomDisk - Erase a disk, writing random data to the media
• secureErase - Securely erase a disk or freespace on a volume
• resizeVolume - Resize a volume, increasing or decreasing its size

See the man page for even more!

Copy and paste inside a shell script

The Terminal application supports copy and paste from the Edit menu or using the standard keyboard shortcuts. In addition, two native Apple command line programs, pbcopy and pbpaste, allow you to copy and paste text from the clipboard within a shell script.

pbcopy takes standard in and places it in the specified pasteboard (the clipboard). For example, to capture the output of the ls -1 command, pipe the output to pbcopy:

ls -1 | pbcopy

Now, the results can be pasted into any program that supports the clipboard.

pbpaste writes the contents of the pasteboard (the clipboard) to standard out. For example, to paste the contents of the clipboard into a text file called "output.txt", use:

pbpaste > output.txt

Defaults -- setting preferences from the command line

The defaults program allows users to read, write, and delete Mac OS X user preferences from the command line. OS X applications use the defaults system to record user preferences and other information that must be maintained when the applications aren't running. Setting preferences is usually done (more easily) from the GUI, but some options are not available in the GUI.

Property lists

User preferences and application settings are stored in files called property lists (with an extension of .plist). These files can store binary values, so they can sometimes look meaningless if you view the raw file. Aside from the Preferences dialogs, property list files can be updated with the Property List Editor application (part of the developer tools), or with the defaults command line program.

User defaults belong to domains, which typically correspond to individual applications. For example, the domain for Firefox is org.mozilla.firefox with the settings saved in a file called org.mozilla.firefox.plist. Each domain has a dictionary of keys and values representing its defaults; for example, "Default Font" = "Helvetica". Keys are always strings, but values can be complex data structures comprising arrays, dictionaries, strings, and binary data.

User plist files are saved in ~/Library/Preferences/, while system wide plist files are stored in /Library/Preferences/.

When you update a setting using defaults, it only affects the current user.

View preferences

To see all user preferences, use:
defaults read

The output list will be very long because every setting for every application will be printed.

To see all Firefox user preferences, use:
defaults read org.mozilla.firefox

Note that Firefox stores most of it's settings in Firefox specific files, and not in Apple's defaults system. The same is true in Windows where most of the Firefox settings are not in the Windows registry.

Update or delete a preference

To update a key use:
defaults write domain { key 'value' }

Of course, you have to know the name of the key and the acceptable values you can use ahead of time.

To delete a key use:
defaults delete domain key

Nvram

The nvram utility is used to view and set Open Firmware NVRAM variables. Open Firmware, (aka non-volatile RAM), controls system boot options. To change a variable, you must run nvram as root.

Viewing Open Firmware variables

There are about 50 Open Firmware variables. To view all variables, use:
nvram -p

Setting Open Firmware variables

You must be root to set a variable. To set a variable, assign a value with an equals sign:
nvram scroll-lock=false

After setting a variable, they are saved during the next clean restart or shutdown.

To set multiple variables at once, list them in a text file in variable=value format, one per line.
Then, feed them all to nvram with:
nvram -f text-file

You can also create new variables by simply assigning a value to a new variable name. It won't have any meaning to the system unless some application reads it, but it is a nice option.

Deleting Open Fireware variables

To delete a variable, use:
nvram -d variable

Sysctl

Sysctl is a kernel utility that allows you to query and set kernel variables. It is rare that you need to tinker with a running OS X kernel, but special applications may require it.

Viewing kernel variables

There are over 100 kernel state variables. To view a specific variable, use:
sysctl variable-name

To view all variables, use:
sysctl -a

Setting kernel variables

You must be root to set a kernel variable. To set a kernel variable, use the -w switch:
sysctl -w variable-name=value

If you are running OS X as a file server and have a lot of people connecting and sharing files, you might want to increase the maximum number of open files allowed by the kernel. This can be tuned using the kern.maxfiles variable. The default value is 12288. To increase it to 50000, use:
sysctl -w kern.maxfiles=50000

Diagnostic command line tools

OS X comes with a number of command line tools to measure system performance and individual application performance. Using these tools, you can track down bottlenecks and discover misbehaving applications -- those acting like rude, thoughtless, little pigs.

top

The top program shows a full screen of stats on overall CPU usage, memory usage, load average, and how many resources each process is using. The default mode for top is an interactive display that updates once per second. Applications are sorted by CPU usage in descending order. To quit while in interactive mode, type q.

Processes:  64 total, 3 running, 1 stuck, 60 sleeping... 204 threads   16:21:14
Load Avg:  0.52, 0.43, 0.25     CPU usage:  9.3% user, 22.9% sys, 67.8% idle
SharedLibs: num =   27, resident = 11.0M code, 1.02M data, 3.44M LinkEdit
MemRegions: num = 11872, resident =  388M + 9.48M private,  111M shared
PhysMem:   117M wired,  393M active,  374M inactive,  885M used,  138M free
VM: 5.57G + 21.5M   284085(0) pageins, 19704(0) pageouts

 PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
23002 mdimport     0.1%  0:00.38   4    62    52   996K  4.27M  3.01M  40.0M
22998 top         12.3%  0:24.47   1    19    22   476K   476K   900K  27.0M
22977 lookupd      0.0%  0:00.09   2    34    37   400K  1.06M  1.16M  28.5M
22971 mdimport     0.0%  0:00.31   3    60    46   864K  3.37M  2.76M  39.0M
19117 firefox-bi  10.1% 12:34:42  10   196   781   143M  59.6M   155M   983M
18711 bash         0.0%  0:00.14   1    14    17   216K   884K   748K  27.2M
18710 login        0.0%  0:00.02   1    16    37   124K   516K   460K  26.9M
17409 httpd        0.0%  0:00.00   1    11    87   104K  2.47M   476K  31.9M
7211 usbmuxd      0.0%  0:00.02   2    23    23   164K   436K   284K  27.0M
7193 iTunesHelp   0.0%  0:01.38   2    55    99   608K  4.67M  1.55M   108M
4618 Terminal     4.2% 27:15.18  10   173   228  3.91M+ 16.5M+ 9.18M+  135M+
 717 AppleSpell   0.0%  0:00.28   1    32    36   284K  2.06M   860K  37.7M
 712 System Eve   0.0%  0:03.10   1    62   109   928K  7.02M  1.30M   109M
 682 cupsd        0.0%  0:58.79   2    30    25   568K  1.03M   844K  27.8M
 380 DashboardC   0.0%  0:56.12   3    78   160  3.59M  6.57M  4.13M   116M
 379 DashboardC   0.0%  0:07.55   4    86   158  5.75M  6.66M  6.56M   115M

Top can also be run in batch mode where it writes output to the terminal a set number of times, then quits. This is ideal for feeding data to a parsing program for generating your own stats or filtering. For example, to show top output once, use:

top -l 1
note: top minus el one

To sort the output by CPU usage, use:

top -l 1 -o cpu

To sort the output by resident memory usage (physical memory), use:

top -l 1 -o rsize

To sort the output by virtual memory usage (total memory), use:

top -l 1 -o vsize

fs_usage

Report system calls and page faults related to filesystem activity in real-time. It is a continuous display and if unfiltered, is overwhelming. It is usually best to limit the output to a single process or to watch network activity. It must be run as root.

To limit the output to a process or command name, include it at the end of the command:

fs_usage 19117

To limit the output to just network related system calls, use:

fs_usage -f network

To limit the output to just filesystem related system calls, use:

fs_usage -f filesys

See also lsof

vm_stat

The vm_stat program displays Mach (the kernel) virtual memory statistics. It is useful to seeing if your system is memory starved.

Here is the output from vm_stat:

Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                    63416.
Pages active:                 103005.
Pages inactive:                68102.
Pages wired down:              27621.
"Translation faults":      308932364.
Pages copy-on-write:         7961115.
Pages zero filled:         204818181.
Pages reactivated:           1571266.
Pageins:                      279532.
Pageouts:                      19704.
Object cache: 1693448 hits of 4581511 lookups (36% hit rate)

If Pages free is low, you need more memory to run all your applications.

You can add an interval (in seconds) as a parameter to vm_stat and it will show your memory and paging activity in real time. For example, vm_stat 1 will output a new line every second -- useful to see if memory is taking a beating with a lot Pageins/Pageouts, but it won't pinpoint an offending application.

30 days with JFS

The Journaled File System (JFS) is a little known file system open sourced by IBM in 1999 and available in the official kernel sources since 2002. It originated inside IBM as the standard file system on the AIX line of UNIX servers, and later OS/2. IBM ships JFS2 with AIX today. Despite its pedigree, JFS has not received the publicity or widespread usage of other Linux file systems like EXT2/3 and ReiserFS. To learn more about JFS, I installed it as my root file system.

note: originally published September 14, 2007 on linux.com

Features

I installed Slackware 12 on a laptop choosing JFS as the file system during installation. I performed no special partitioning and created one JFS file system to hold everything. Installation was uneventful and the system booted normally from Grub. Not all distributions offer JFS as an install option, and some may not have JFS compiled into their default kernels. While Fedora and SUSE can use JFS, they both default to EXT3. Slackware, Debian, Ubuntu, and their derivatives are good choices if you want to try JFS.

One of the first things I noticed was the absence of a lost+found directory, a relic of lesser file systems.

JFS is a fully 64-bit file system. With the default block size of 4 KB, it supports a maximum file system size of 4 petabytes (less if smaller block sizes are used). The minimum file system size supported is 16 MB. The JFS transaction log has a default size of 0.4% of the aggregate size, rounded up to a megabyte boundary. The maximum size of the log is 32 MB. One interesting aspect of the layout on disk is the fsck working space, a small area allocated within the file system for keeping track of block allocation if there is not enough RAM to track a large file system at boot time.

JFS dynamically allocates space for disk inodes, freeing the space when it is no longer required. This eliminates the possibility of running out of inodes due to a large number of small files. As far as I can tell, JFS is the only file system in the kernel with this feature. For performance and efficiency, the contents of small directories are stored within the directory's inode. Up to eight entries are stored in-line within the inode, excluding the self [.] and parent [..] entries. Larger directories use a B+tree keyed on name for faster retrieval. Internally, extents are used to allocate blocks to files, leading to compact, efficient use of space even as files grow in size. This is also available in XFS, and is a major new feature in EXT4.

JFS supports both sparse and dense files. Sparse files allow data to be written to random locations within a file without writing intervening file blocks. JFS reports the file size as the largest used block, while only allocating actually used blocks. Sparse files are useful for applications that require a large logical space but only use a portion of the space. With dense files, blocks are allocated to fill the entire file size, whether data is written to them or not.

In addition to the standard permissions, JFS supports basic extended attributes, such as the immutable (i) and append-only (a) attributes. I was able to successfully set and test them with the lsattr and chattr programs. I could not find definitive information on access control list support under Linux.

Logging

The main design goal of JFS was to provide fast crash recovery for large file systems, avoiding the long file system check (fsck) times of older UNIX file systems. That was also the primary goal of file systems like EXT3 and ReiserFS. Unlike EXT3, journaling was not an add-on to JFS, but baked into the design from the start. For high performance applications, the JFS transaction log file can be created on an external volume if specified when the file system is first created.

JFS only logs operations on meta-data, maintaining the consistency of the file system structure, but not necessarily the data. A crash might result in stale data, but the files should remain consistent and usable.

Here is a list of the file system operations logged by JFS:

• File creation (create)
• Linking (link)
• Making directory (mkdir)
• Making node (mknod)
• Removing file (unlink)
• Rename (rename)
• Removing directory (rmdir)
• Symbolic link (symlink)
• Truncating regular file

Utilities

A suite of utilities is provided to manage JFS file systems. You must be the root user to use them.

Utility	Description
jfs_debugfs	shell based JFS file system editor, allows changes to the ACL, uid/gid. mode, time, etc. You can also alter data on disk, but only by entering hex strings, not the most efficient way to edit a file.
jfs_fsck	replay the JFS transaction log, check and repair a JFS device, should on ly be run on an unmounted or read only file system, run automatically at boot.
jfs_fscklog	extract a JFS fsck service log into a file. jfs_fscklog -e /dev/hd a6 extracts the binary log to file fscklog.new, to view, use jfs_fscklog -d fscklog.new
jfs_logdump	dump the journal log, a plaint text file that shows data on each transac tion in the log file.
jfs_mkfs	create a JFS formatted partition, use the -j journal_device option to create an external journal (1.0.18 or later)
jfs_tune	adjust tunable file system parameters on JFS. I didn't find options that looked like they might improve performance, the -l option lists th e superblock info

Here is what a dump of the superblock information looks like:

root@slackt41:~# jfs_tune -l /dev/hda6
jfs_tune version 1.1.11, 05-Jun-2006

JFS filesystem superblock:

JFS magic number:       'JFS1'
JFS version:            1
JFS state:              mounted
JFS flags:              JFS_LINUX  JFS_COMMIT  JFS_GROUPCOMMIT  JFS_INLINELOG
Aggregate block size:   4096 bytes
Aggregate size:         12239720 blocks
Physical block size:    512 bytes
Allocation group size:  16384 aggregate blocks
Log device number:      0x306
Filesystem creation:    Wed Jul 11 01:52:42 2007
Volume label:           ''

Crash Testing

White papers and man pages are no substitute for the harsh reality of a server room. To test the recovery capabilities of JFS, I started crashing my system (forced power off) with increasing workloads. I repeated each crash twice to see if my results were consistent.

Crash workload	Recovery
console (no X) running text editor with one open file	about 2 seconds to replay the journal log, changes I had not saved in th e editor were missing but the file was intact
X window system with KDE, gimp, nvu, and text editor in xte rm all with open files	about 2 seconds to replay the journal log, all open files were intact, u nsaved changes were missing
X window system with KDE, gimp, nvu, and text editor all with open files , plus a shell script that inserted records into a MySQL (ISAM) table. The scri pt I wrote was an infinite loop and I let it run for a couple of minutes to make sure some records were flushed to disk.	about 3 seconds to replay the journal log, all open files intact, databa se intact with a few thousand records inserted, but the timestamp on the table f ile had been rolled back one minute.

In all cases, these boot messages appeared...

**Phase 0 - Replay Journal Log
-|----- (spinner appeared for a couple of seconds, then went away)
Filesystem is clean

Throughout the crash testing, no file system corruption occurred and the longest log replay time I experienced was about 3 seconds.

Conclusion

While my improvised crash tests were not a good simulation a busy server, JFS did hold up well, and recovery time was fast. All file level applications I tested, like tar and rsync, worked flawlessly and lower level programs like truecrypt also worked as expected. After 30 days of kicking and prodding, this gave me a high level of confidence in JFS and I am content trusting my data to it. JFS may not have been marketed as effectively as others, but is a solid choice in the long list of quality Linux file systems.

MacPorts, Up and Running

MacPorts is a system designed to let you easily install and manage free, open source software on OS X. Until late in 2006, the project was named DarwinPorts. There are disk image downloads available for 10.3 (Panther), 10.4 (Tiger), and 10.5 (Leopard). If you have a different version, you can download the source code and compile it.

updated for MacPorts 1.6 and Leopard on January 16, 2008

The Ports concept

If you are new the "ports" concept, the way the system works may be hard to grasp. The ports system was developed in the BSD community as a way to easily install and manage software on any BSD system. It was designed to automate the manual process programmers followed to port software from one Unix-like system to another. Most of the software in the ports repository is written in the C language and requires a C compiler.

When a particular package is requested, the ports system checks the version of the system it is running on, the version of the available source code, then downloads the source, applies any patches needed for it run on the local system, compiles it, and installs it. The design of the ports system only requires package maintainers to write scripts that handle any differences in the local system (like library versions, hardware architecture, compiler versions, etc.). The heavy lifting is done by the ports system itself.

This system is ingenious and works well in most situations, but you may occasionally run into a package that doesn't work. If that happens, it is usually because the source program was recently updated and the port scripts haven't caught up.

Universal ports

An Apple enhancement to the original ports system allows some software to be built (compiled) as universal binaries. This option may be desirable if you share binaries between Intel and PPC Macs, or plan to upgrade soon and don't want to install your ports again.

Installing MacPorts

According to the MacPorts wiki, there are three critical steps that must be done prior to installing MacPorts:

1. Install Xcode Tools
2. Install XWindows (X11)
3. Set the shell environment

Both Xcode Tools (which includes the compiler), and X11 are available on the Mac DVDs. They can also be downloaded from the Apple developer web site. I had already installed Xcode and XWindows, so the only thing I needed to do was modify my $HOME/.bash_login file to include the /opt/local directories in my PATH and set the DISPLAY variable with "export DISPLAY=:0.0". See the wiki page for additional details.

After setting my shell environment, I downloaded version 1.6 of the MacPorts disk image for 10.5 (Leopard) from the download area. I mounted the disk image and ran the .pkg installer to install MacPorts. It completed with no errors.

If you want to tweak the port system configuration, the default location of the global config file is /opt/local/etc/macports/macports.conf.

Getting started

Once the ports system was installed, I followed the wiki recommendation and updated MacPorts itself with this Terminal command:

sudo port -d selfupdate

Next, I installed a simple program to see what kind of output the system produced. Here is the output from installing dos2unix, a program that changes the line endings of a text file from DOS to UNIX format.

sudo port install dos2unix
Password:
--->  Fetching dos2unix
--->  Attempting to fetch dos2unix-3.1.tar.gz from http://fresh.t-systems-sfr.com/linux/src/
--->  Verifying checksum(s) for dos2unix
--->  Extracting dos2unix
--->  Configuring dos2unix
--->  Building dos2unix with target all
--->  Staging dos2unix into destroot
--->  Installing dos2unix 3.1_0
--->  Activating dos2unix 3.1_0
--->  Cleaning dos2unix

When the port command finished, I had a freshly compiled dos2unix program sitting in /opt/local/bin/. Everything is nicely isolated in /opt/local/ to prevent any port program from interfering with the original Apple system.

One of more powerful features of ports is that it understands program dependencies. If you ask it to install a program that requires other programs or libraries to run, the ports system first installs the required components, then completes your request. For example, I wanted to install wget, a useful command line tool to download files or mirror web sites. I started the install with:

sudo port install wget

After starting it, I walked away. When I came back 30 minutes later, ports was still still installing dependencies. There were five dependencies that got installed first:

expat
libiconv
gettext
zlib
openssl

If you start a port install and your system seems sluggish, check to see if ports is still downloading and compiling dependencies.

Uninstalling MacPorts

Because of the way MacPorts files are isolated, it can be easily removed. To uninstall both the MacPorts system and all ports installed, the wiki lists the following files and directories to delete:

/opt/local/
/Applications/MacPorts/
/Library/Tcl/macports1.0/
/Library/LaunchDaemons/org.macports.*
/Library/StartupItems/DarwinPortsStartup

To my surprise, my installation did not include these files and directories:

/Applications/MacPorts/
/Library/LaunchDaemons/org.macports.*
/Library/StartupItems/DarwinPortsStartup

Since the system appears to be fully functional, it is possible that this part of the wiki is slightly out of date.

Update January 10, 2008: Seth Milliken wrote in and noted that these directories are only created if you install a port that uses them, e.g. installing a Cocoa app will create /Applications/MacPorts/.

Common port commands

Here are some commands you may find useful while learning to use the ports system.

Command	Result
port install package	installs a package
port uninstall package	uninstalls a package
port installed	shows all ports currently installed
port uninstalled	shows all ports not currently installed, you may want to pipe the output of this command to grep to limit the output
port search regex	search for a port whose name matches the regex
port info package	shows meta information on a package
port list	shows the latest version of all available ports

See the port man page for additional options.

Filling in the holes

MacPorts can sometimes provide a software solution the default Apple software does not. For me, that usually means a command line or system oriented tool, but there are plenty of GUI applications available, too. The MacPorts team has done a nice job "porting" the ports system to the Mac. It is one of the most popular features of BSD, and one of the reasons I am running a Mac today.