Copy and paste inside a shell script

The Terminal application supports copy and paste from the Edit menu or using the standard keyboard shortcuts. In addition, two native Apple command line programs, pbcopy and pbpaste, allow you to copy and paste text from the clipboard within a shell script.

pbcopy takes standard in and places it in the specified pasteboard (the clipboard). For example, to capture the output of the ls -1 command, pipe the output to pbcopy:

ls -1 | pbcopy

Now, the results can be pasted into any program that supports the clipboard.

pbpaste writes the contents of the pasteboard (the clipboard) to standard out. For example, to paste the contents of the clipboard into a text file called "output.txt", use:

pbpaste > output.txt

Defaults -- setting preferences from the command line

The defaults program allows users to read, write, and delete Mac OS X user preferences from the command line. OS X applications use the defaults system to record user preferences and other information that must be maintained when the applications aren't running. Setting preferences is usually done (more easily) from the GUI, but some options are not available in the GUI.

Property lists

User preferences and application settings are stored in files called property lists (with an extension of .plist). These files can store binary values, so they can sometimes look meaningless if you view the raw file. Aside from the Preferences dialogs, property list files can be updated with the Property List Editor application (part of the developer tools), or with the defaults command line program.

User defaults belong to domains, which typically correspond to individual applications. For example, the domain for Firefox is org.mozilla.firefox with the settings saved in a file called org.mozilla.firefox.plist. Each domain has a dictionary of keys and values representing its defaults; for example, "Default Font" = "Helvetica". Keys are always strings, but values can be complex data structures comprising arrays, dictionaries, strings, and binary data.

User plist files are saved in ~/Library/Preferences/, while system wide plist files are stored in /Library/Preferences/.

When you update a setting using defaults, it only affects the current user.

View preferences

To see all user preferences, use:
defaults read

The output list will be very long because every setting for every application will be printed.

To see all Firefox user preferences, use:
defaults read org.mozilla.firefox

Note that Firefox stores most of it's settings in Firefox specific files, and not in Apple's defaults system. The same is true in Windows where most of the Firefox settings are not in the Windows registry.

Update or delete a preference

To update a key use:
defaults write domain { key 'value' }

Of course, you have to know the name of the key and the acceptable values you can use ahead of time.

To delete a key use:
defaults delete domain key

Nvram

The nvram utility is used to view and set Open Firmware NVRAM variables. Open Firmware, (aka non-volatile RAM), controls system boot options. To change a variable, you must run nvram as root.

Viewing Open Firmware variables

There are about 50 Open Firmware variables. To view all variables, use:
nvram -p

Setting Open Firmware variables

You must be root to set a variable. To set a variable, assign a value with an equals sign:
nvram scroll-lock=false

After setting a variable, they are saved during the next clean restart or shutdown.

To set multiple variables at once, list them in a text file in variable=value format, one per line.
Then, feed them all to nvram with:
nvram -f text-file

You can also create new variables by simply assigning a value to a new variable name. It won't have any meaning to the system unless some application reads it, but it is a nice option.

Deleting Open Fireware variables

To delete a variable, use:
nvram -d variable

Sysctl

Sysctl is a kernel utility that allows you to query and set kernel variables. It is rare that you need to tinker with a running OS X kernel, but special applications may require it.

Viewing kernel variables

There are over 100 kernel state variables. To view a specific variable, use:
sysctl variable-name

To view all variables, use:
sysctl -a

Setting kernel variables

You must be root to set a kernel variable. To set a kernel variable, use the -w switch:
sysctl -w variable-name=value

If you are running OS X as a file server and have a lot of people connecting and sharing files, you might want to increase the maximum number of open files allowed by the kernel. This can be tuned using the kern.maxfiles variable. The default value is 12288. To increase it to 50000, use:
sysctl -w kern.maxfiles=50000

Diagnostic command line tools

OS X comes with a number of command line tools to measure system performance and individual application performance. Using these tools, you can track down bottlenecks and discover misbehaving applications -- those acting like rude, thoughtless, little pigs.

top

The top program shows a full screen of stats on overall CPU usage, memory usage, load average, and how many resources each process is using. The default mode for top is an interactive display that updates once per second. Applications are sorted by CPU usage in descending order. To quit while in interactive mode, type q.

Processes:  64 total, 3 running, 1 stuck, 60 sleeping... 204 threads   16:21:14
Load Avg:  0.52, 0.43, 0.25     CPU usage:  9.3% user, 22.9% sys, 67.8% idle
SharedLibs: num =   27, resident = 11.0M code, 1.02M data, 3.44M LinkEdit
MemRegions: num = 11872, resident =  388M + 9.48M private,  111M shared
PhysMem:   117M wired,  393M active,  374M inactive,  885M used,  138M free
VM: 5.57G + 21.5M   284085(0) pageins, 19704(0) pageouts

 PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
23002 mdimport     0.1%  0:00.38   4    62    52   996K  4.27M  3.01M  40.0M
22998 top         12.3%  0:24.47   1    19    22   476K   476K   900K  27.0M
22977 lookupd      0.0%  0:00.09   2    34    37   400K  1.06M  1.16M  28.5M
22971 mdimport     0.0%  0:00.31   3    60    46   864K  3.37M  2.76M  39.0M
19117 firefox-bi  10.1% 12:34:42  10   196   781   143M  59.6M   155M   983M
18711 bash         0.0%  0:00.14   1    14    17   216K   884K   748K  27.2M
18710 login        0.0%  0:00.02   1    16    37   124K   516K   460K  26.9M
17409 httpd        0.0%  0:00.00   1    11    87   104K  2.47M   476K  31.9M
7211 usbmuxd      0.0%  0:00.02   2    23    23   164K   436K   284K  27.0M
7193 iTunesHelp   0.0%  0:01.38   2    55    99   608K  4.67M  1.55M   108M
4618 Terminal     4.2% 27:15.18  10   173   228  3.91M+ 16.5M+ 9.18M+  135M+
 717 AppleSpell   0.0%  0:00.28   1    32    36   284K  2.06M   860K  37.7M
 712 System Eve   0.0%  0:03.10   1    62   109   928K  7.02M  1.30M   109M
 682 cupsd        0.0%  0:58.79   2    30    25   568K  1.03M   844K  27.8M
 380 DashboardC   0.0%  0:56.12   3    78   160  3.59M  6.57M  4.13M   116M
 379 DashboardC   0.0%  0:07.55   4    86   158  5.75M  6.66M  6.56M   115M

Top can also be run in batch mode where it writes output to the terminal a set number of times, then quits. This is ideal for feeding data to a parsing program for generating your own stats or filtering. For example, to show top output once, use:

top -l 1
note: top minus el one

To sort the output by CPU usage, use:

top -l 1 -o cpu

To sort the output by resident memory usage (physical memory), use:

top -l 1 -o rsize

To sort the output by virtual memory usage (total memory), use:

top -l 1 -o vsize

fs_usage

Report system calls and page faults related to filesystem activity in real-time. It is a continuous display and if unfiltered, is overwhelming. It is usually best to limit the output to a single process or to watch network activity. It must be run as root.

To limit the output to a process or command name, include it at the end of the command:

fs_usage 19117

To limit the output to just network related system calls, use:

fs_usage -f network

To limit the output to just filesystem related system calls, use:

fs_usage -f filesys

See also lsof

vm_stat

The vm_stat program displays Mach (the kernel) virtual memory statistics. It is useful to seeing if your system is memory starved.

Here is the output from vm_stat:

Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                    63416.
Pages active:                 103005.
Pages inactive:                68102.
Pages wired down:              27621.
"Translation faults":      308932364.
Pages copy-on-write:         7961115.
Pages zero filled:         204818181.
Pages reactivated:           1571266.
Pageins:                      279532.
Pageouts:                      19704.
Object cache: 1693448 hits of 4581511 lookups (36% hit rate)

If Pages free is low, you need more memory to run all your applications.

You can add an interval (in seconds) as a parameter to vm_stat and it will show your memory and paging activity in real time. For example, vm_stat 1 will output a new line every second -- useful to see if memory is taking a beating with a lot Pageins/Pageouts, but it won't pinpoint an offending application.

30 days with JFS

The Journaled File System (JFS) is a little known file system open sourced by IBM in 1999 and available in the official kernel sources since 2002. It originated inside IBM as the standard file system on the AIX line of UNIX servers, and later OS/2. IBM ships JFS2 with AIX today. Despite its pedigree, JFS has not received the publicity or widespread usage of other Linux file systems like EXT2/3 and ReiserFS. To learn more about JFS, I installed it as my root file system.

note: originally published September 14, 2007 on linux.com

Features

I installed Slackware 12 on a laptop choosing JFS as the file system during installation. I performed no special partitioning and created one JFS file system to hold everything. Installation was uneventful and the system booted normally from Grub. Not all distributions offer JFS as an install option, and some may not have JFS compiled into their default kernels. While Fedora and SUSE can use JFS, they both default to EXT3. Slackware, Debian, Ubuntu, and their derivatives are good choices if you want to try JFS.

One of the first things I noticed was the absence of a lost+found directory, a relic of lesser file systems.

JFS is a fully 64-bit file system. With the default block size of 4 KB, it supports a maximum file system size of 4 petabytes (less if smaller block sizes are used). The minimum file system size supported is 16 MB. The JFS transaction log has a default size of 0.4% of the aggregate size, rounded up to a megabyte boundary. The maximum size of the log is 32 MB. One interesting aspect of the layout on disk is the fsck working space, a small area allocated within the file system for keeping track of block allocation if there is not enough RAM to track a large file system at boot time.

JFS dynamically allocates space for disk inodes, freeing the space when it is no longer required. This eliminates the possibility of running out of inodes due to a large number of small files. As far as I can tell, JFS is the only file system in the kernel with this feature. For performance and efficiency, the contents of small directories are stored within the directory's inode. Up to eight entries are stored in-line within the inode, excluding the self [.] and parent [..] entries. Larger directories use a B+tree keyed on name for faster retrieval. Internally, extents are used to allocate blocks to files, leading to compact, efficient use of space even as files grow in size. This is also available in XFS, and is a major new feature in EXT4.

JFS supports both sparse and dense files. Sparse files allow data to be written to random locations within a file without writing intervening file blocks. JFS reports the file size as the largest used block, while only allocating actually used blocks. Sparse files are useful for applications that require a large logical space but only use a portion of the space. With dense files, blocks are allocated to fill the entire file size, whether data is written to them or not.

In addition to the standard permissions, JFS supports basic extended attributes, such as the immutable (i) and append-only (a) attributes. I was able to successfully set and test them with the lsattr and chattr programs. I could not find definitive information on access control list support under Linux.

Logging

The main design goal of JFS was to provide fast crash recovery for large file systems, avoiding the long file system check (fsck) times of older UNIX file systems. That was also the primary goal of file systems like EXT3 and ReiserFS. Unlike EXT3, journaling was not an add-on to JFS, but baked into the design from the start. For high performance applications, the JFS transaction log file can be created on an external volume if specified when the file system is first created.

JFS only logs operations on meta-data, maintaining the consistency of the file system structure, but not necessarily the data. A crash might result in stale data, but the files should remain consistent and usable.

Here is a list of the file system operations logged by JFS:

• File creation (create)
• Linking (link)
• Making directory (mkdir)
• Making node (mknod)
• Removing file (unlink)
• Rename (rename)
• Removing directory (rmdir)
• Symbolic link (symlink)
• Truncating regular file

Utilities

A suite of utilities is provided to manage JFS file systems. You must be the root user to use them.

Utility	Description
jfs_debugfs	shell based JFS file system editor, allows changes to the ACL, uid/gid. mode, time, etc. You can also alter data on disk, but only by entering hex strings, not the most efficient way to edit a file.
jfs_fsck	replay the JFS transaction log, check and repair a JFS device, should on ly be run on an unmounted or read only file system, run automatically at boot.
jfs_fscklog	extract a JFS fsck service log into a file. jfs_fscklog -e /dev/hd a6 extracts the binary log to file fscklog.new, to view, use jfs_fscklog -d fscklog.new
jfs_logdump	dump the journal log, a plaint text file that shows data on each transac tion in the log file.
jfs_mkfs	create a JFS formatted partition, use the -j journal_device option to create an external journal (1.0.18 or later)
jfs_tune	adjust tunable file system parameters on JFS. I didn't find options that looked like they might improve performance, the -l option lists th e superblock info

Here is what a dump of the superblock information looks like:

root@slackt41:~# jfs_tune -l /dev/hda6
jfs_tune version 1.1.11, 05-Jun-2006

JFS filesystem superblock:

JFS magic number:       'JFS1'
JFS version:            1
JFS state:              mounted
JFS flags:              JFS_LINUX  JFS_COMMIT  JFS_GROUPCOMMIT  JFS_INLINELOG
Aggregate block size:   4096 bytes
Aggregate size:         12239720 blocks
Physical block size:    512 bytes
Allocation group size:  16384 aggregate blocks
Log device number:      0x306
Filesystem creation:    Wed Jul 11 01:52:42 2007
Volume label:           ''

Crash Testing

White papers and man pages are no substitute for the harsh reality of a server room. To test the recovery capabilities of JFS, I started crashing my system (forced power off) with increasing workloads. I repeated each crash twice to see if my results were consistent.

Crash workload	Recovery
console (no X) running text editor with one open file	about 2 seconds to replay the journal log, changes I had not saved in th e editor were missing but the file was intact
X window system with KDE, gimp, nvu, and text editor in xte rm all with open files	about 2 seconds to replay the journal log, all open files were intact, u nsaved changes were missing
X window system with KDE, gimp, nvu, and text editor all with open files , plus a shell script that inserted records into a MySQL (ISAM) table. The scri pt I wrote was an infinite loop and I let it run for a couple of minutes to make sure some records were flushed to disk.	about 3 seconds to replay the journal log, all open files intact, databa se intact with a few thousand records inserted, but the timestamp on the table f ile had been rolled back one minute.

In all cases, these boot messages appeared...

**Phase 0 - Replay Journal Log
-|----- (spinner appeared for a couple of seconds, then went away)
Filesystem is clean

Throughout the crash testing, no file system corruption occurred and the longest log replay time I experienced was about 3 seconds.

Conclusion

While my improvised crash tests were not a good simulation a busy server, JFS did hold up well, and recovery time was fast. All file level applications I tested, like tar and rsync, worked flawlessly and lower level programs like truecrypt also worked as expected. After 30 days of kicking and prodding, this gave me a high level of confidence in JFS and I am content trusting my data to it. JFS may not have been marketed as effectively as others, but is a solid choice in the long list of quality Linux file systems.

MacPorts, Up and Running

MacPorts is a system designed to let you easily install and manage free, open source software on OS X. Until late in 2006, the project was named DarwinPorts. There are disk image downloads available for 10.3 (Panther), 10.4 (Tiger), and 10.5 (Leopard). If you have a different version, you can download the source code and compile it.

updated for MacPorts 1.6 and Leopard on January 16, 2008

The Ports concept

If you are new the "ports" concept, the way the system works may be hard to grasp. The ports system was developed in the BSD community as a way to easily install and manage software on any BSD system. It was designed to automate the manual process programmers followed to port software from one Unix-like system to another. Most of the software in the ports repository is written in the C language and requires a C compiler.

When a particular package is requested, the ports system checks the version of the system it is running on, the version of the available source code, then downloads the source, applies any patches needed for it run on the local system, compiles it, and installs it. The design of the ports system only requires package maintainers to write scripts that handle any differences in the local system (like library versions, hardware architecture, compiler versions, etc.). The heavy lifting is done by the ports system itself.

This system is ingenious and works well in most situations, but you may occasionally run into a package that doesn't work. If that happens, it is usually because the source program was recently updated and the port scripts haven't caught up.

Universal ports

An Apple enhancement to the original ports system allows some software to be built (compiled) as universal binaries. This option may be desirable if you share binaries between Intel and PPC Macs, or plan to upgrade soon and don't want to install your ports again.

Installing MacPorts

According to the MacPorts wiki, there are three critical steps that must be done prior to installing MacPorts:

1. Install Xcode Tools
2. Install XWindows (X11)
3. Set the shell environment

Both Xcode Tools (which includes the compiler), and X11 are available on the Mac DVDs. They can also be downloaded from the Apple developer web site. I had already installed Xcode and XWindows, so the only thing I needed to do was modify my $HOME/.bash_login file to include the /opt/local directories in my PATH and set the DISPLAY variable with "export DISPLAY=:0.0". See the wiki page for additional details.

After setting my shell environment, I downloaded version 1.6 of the MacPorts disk image for 10.5 (Leopard) from the download area. I mounted the disk image and ran the .pkg installer to install MacPorts. It completed with no errors.

If you want to tweak the port system configuration, the default location of the global config file is /opt/local/etc/macports/macports.conf.

Getting started

Once the ports system was installed, I followed the wiki recommendation and updated MacPorts itself with this Terminal command:

sudo port -d selfupdate

Next, I installed a simple program to see what kind of output the system produced. Here is the output from installing dos2unix, a program that changes the line endings of a text file from DOS to UNIX format.

sudo port install dos2unix
Password:
--->  Fetching dos2unix
--->  Attempting to fetch dos2unix-3.1.tar.gz from http://fresh.t-systems-sfr.com/linux/src/
--->  Verifying checksum(s) for dos2unix
--->  Extracting dos2unix
--->  Configuring dos2unix
--->  Building dos2unix with target all
--->  Staging dos2unix into destroot
--->  Installing dos2unix 3.1_0
--->  Activating dos2unix 3.1_0
--->  Cleaning dos2unix

When the port command finished, I had a freshly compiled dos2unix program sitting in /opt/local/bin/. Everything is nicely isolated in /opt/local/ to prevent any port program from interfering with the original Apple system.

One of more powerful features of ports is that it understands program dependencies. If you ask it to install a program that requires other programs or libraries to run, the ports system first installs the required components, then completes your request. For example, I wanted to install wget, a useful command line tool to download files or mirror web sites. I started the install with:

sudo port install wget

After starting it, I walked away. When I came back 30 minutes later, ports was still still installing dependencies. There were five dependencies that got installed first:

expat
libiconv
gettext
zlib
openssl

If you start a port install and your system seems sluggish, check to see if ports is still downloading and compiling dependencies.

Uninstalling MacPorts

Because of the way MacPorts files are isolated, it can be easily removed. To uninstall both the MacPorts system and all ports installed, the wiki lists the following files and directories to delete:

/opt/local/
/Applications/MacPorts/
/Library/Tcl/macports1.0/
/Library/LaunchDaemons/org.macports.*
/Library/StartupItems/DarwinPortsStartup

To my surprise, my installation did not include these files and directories:

/Applications/MacPorts/
/Library/LaunchDaemons/org.macports.*
/Library/StartupItems/DarwinPortsStartup

Since the system appears to be fully functional, it is possible that this part of the wiki is slightly out of date.

Update January 10, 2008: Seth Milliken wrote in and noted that these directories are only created if you install a port that uses them, e.g. installing a Cocoa app will create /Applications/MacPorts/.

Common port commands

Here are some commands you may find useful while learning to use the ports system.

Command	Result
port install package	installs a package
port uninstall package	uninstalls a package
port installed	shows all ports currently installed
port uninstalled	shows all ports not currently installed, you may want to pipe the output of this command to grep to limit the output
port search regex	search for a port whose name matches the regex
port info package	shows meta information on a package
port list	shows the latest version of all available ports

See the port man page for additional options.

Filling in the holes

MacPorts can sometimes provide a software solution the default Apple software does not. For me, that usually means a command line or system oriented tool, but there are plenty of GUI applications available, too. The MacPorts team has done a nice job "porting" the ports system to the Mac. It is one of the most popular features of BSD, and one of the reasons I am running a Mac today.