Introduction to Open Source Scripting on Mac OS X

From : http://developer.apple.com/internet/opensource/opensourcescripting.html

It's really a good article, so I referenced.

 

--------------------------------------------------------------------------------------------------------------------

One of the biggest advantages of Mac OS X's UNIX heritage is the huge range of open source scripting languages available. Scripting languages are often the ideal tool if you want to manipulate text, manage jobs, or link together disparate components without resorting to a compiled language like C++ or Objective-C. Scripting languages are a great tool for system administrators, application developers, and pretty much any user who needs to perform complex or repetitive tasks, because they were invented to solve these types of problems more quickly than can be done with general-purpose languages.

If you're new to the world of open source scripting languages—or just want to brush up on what's unique about Mac OS X—this article will help you get oriented.  Mac OS X bundles the most popular scripting languages out of the box: perl, Python, php, tcl, Ruby—not to mention shells such as bash , ksh , zsh , and csh .  We'll show you how to integrate scripting languages into Apple's Integrated Development Environment (IDE), Xcode , using targets and custom build rules. We'll also discuss ways to safely install newer versions or alternate languages.

AppleScript, the native scripting language for Mac OS X, will not be covered in this article. See the Reference Library section on AppleScript for more information.

How Scripting Is Used

Early scripting languages were often created for specific problem domains. Some were, and still are used to control specific applications. One of the most common uses for scripting languages is to perform batch processing on files. In fact, each of the command shells available on Mac OS X (sh , bash , csh , zsh , and ksh ) has its own language for this purpose. 

In addition to languages specific to shells, there are shell-agnostic scripting languages that perform batch file operations. These languages, such as sed and awk , offer highly specialized string and pattern matching abilities, making them perfect for parsing and altering the contents of files en masse.

Scripting languages have evolved over the years, and sometimes the line between scripting languages and general-purpose programming languages is quite blurry. Languages such as Python are so powerful, they can be used to develop entire applications.

Overview of Open Source Scripting on Mac OS X

Shells and Shell Scripts

Years ago, your sole means for accomplishing tasks—any task—was by typing commands into a shell. The shell was the interface between you and the operating system. The advent of windowing systems brought the choice of using a mouse and graphical user interface, or of using the traditional keyboard-oriented interface, or both.  While use of command shells has diminished somewhat, particularly on systems like Mac OS X, shells are still in use and are not going away any time soon. As usual the Mac comes with a choice of popular UNIX shells, as shown in the table.

Shell Website Notes

bash	bash Home	The "Bourne-again" shell. The GNU version of sh , the original Bourne shell.
csh	csh on freebsd.org	The C shell. Developed at UC Berkeley by Bill Joy, the C shell scripting language shares many features with the C programming language.
tcsh	tcsh Home	The Tenex C Shell. An enhanced but compatible version of csh .
ksh	ksh Home	The Korn shell. Developed by David G. Korn at AT&T Bell Laboratories.
zsh	zsh Home	Zsh was originally written by Paul Falstad at Princeton. It most closely resembles ksh , but includes some features from csh and tcsh as well.
tclsh	Tcl home on SourceForge Tcl Developer Xchange	Tclsh is a shell-like application that reads commands in the Tool Command Language (tcl).

Table 1: Shells included with Mac OS X

The default shell on Mac OS X is the "Bourne-again" shell, or bash . You may configure Terminal.app to use any of the available shells (or any other command shell of your choosing). To configure Terminal.app to use a different shell:

1. Start Terminal.app
2. Choose Terminal > Preferences...
3. Select the Execute this command radio button.
4. In the text field, enter the full path to the shell command you wish to execute.

Note: On Mac OS X, the Bourne Shell, sh, is present, but it is a physical copy of the bash shell executable.

All shells included with Mac OS X are located in the /bin directory. For example, to execute csh instead of bash , enter:

/bin/csh

With all shells, you may enter commands directly at a prompt, or you can automate tasks by creating a shell script file .  Any command you enter at the command line may also be used in a script file. The following simple bash script uses the echo command to print a message in a terminal window:

#!/bin/bash
echo "Hello from a bash script file."

Notice the first line of the script. This is a common feature of all shell scripts no matter which shell you are using. The first line features the #! characters followed by the path to the shell that will execute the script, in this case /bin/bash . This allows you to run scripts scripts written for any shell, no matter what shell your terminal is configured to run. For example, you could run the bash script above, even if you use csh in your terminal window. 

You may create shell scripts using any text file editor. All shell scripts must have their executable bit set. After creating the shell script above in a file called hello.sh , set the executable bit with the chmod command:

chmod +x hello.sh

Typical of all shell languages, the bash shell has a wide range of programming constructs and capabilities. The bash shell features:

• Arithmetic operators
• Looping constructs
• Conditional constructs including a case command 
• User-defined functions 
• Setting and reading variables, including environment variables

The next example is a bash script that copies files into a backup directory. The example demonstrates conditional statements, for loops, and arithmetic expressions. Each numbered comment is called out in more detail, following the code.

#!/bin/bash
# Copy files into a backup folder, skipping directories.

# 1. If the backup directory does not exist, create it.
if [ ! -d ./bak ]
then
 mkdir ./bak
fi

# 2. for each file in the current directory listing
for file in `ls`
 do
 # 3. Skip it if it's a directory...
 if [ -d $file ] 
 then
 echo $file " is a directory."
 else
 # 4. If a backup version already exists, skip past them all and backup a new version.
 # The value of index will be appended to the file name.
 index=0
 if [ -f ./bak/$file.0 ]
 then
 for bakfile in `ls ./bak/$file.*`
	do
 # 5. Increment the value of the index.
	index=`expr $index + 1`
	done
 fi
 echo "Copying " $file
 cp $file ./bak/$file.$index
 fi
 done

1. The if command is used to do conditional testing. Note the spacing on this line—spaces are required before and after the brackets ('[' and ']'). The test looks for the existence of a directory called bak . If it is not there, it is created.
2. The for command iterates over a set of values. In this case the set is the result of a directory listing produced by the ls command. The variable file is the loop variable, and will hold the current value from the set on each iteration.
3. The -d test determines if the file is a directory. If it is, the message "<file> is a directory." is printed. Notice here, and in the if test, the value of the variable file is retrieved by preceding the name with a dollar sign ('$').
4. The next if test looks for an existing version of the file in the backup directory. If one exists, a second for-loop iterates over all versions, to determine the highest version number.
5. The index variable is incremented each time through the loop. The expr command must be used to evaluate the assignment. Without the expr command, the value of index would be set to a literal string instead of a numeric value. Also note the lack of a space between the variable index and the string `expr $index + 1` . If you insert a space, the shell will try to execute index as a command. Shells can be quite temperamental!

Note: You can find a much more capable version of this script, written by Anand Pillai, on the Python Cookbook website.

SED and AWK

Two unusual commands with a long UNIX history are sed and awk . The sed command stands for stream editor . With sed you can perform text editing tasks you would normally perform in a full editor like vi. The sed command is nondestructive; it reads its input file one line at a time, and sends its output to the screen by default. Therefore you cannot use sed to directly modify a file, you can only save its output by redirecting its output into a new file.

Awk is a programming language created by Alfred Aho, Peter Weinberger and Brian Kernighan (hence the name, awk ). The awk command is similar to but much more sophisticated than sed . Awk scans its input, one line at a time, looking for lines that match a specified pattern, and performing a set of operations on those lines that match. You may have a pattern with no operations. In this case awk simply sends all lines that match to the screen by default. If you have an operation with no pattern, the action is applied to all input lines. 

While sed gives you basic editing commands such as line deletion and text substitution, awk provides this capability and more. The awk language has commands for

• Formatted output
• Relational operators
• Arithmetic parameters
• Conditional expressions
• Loop constructs

Both Sed and Awk have regular expression languages to match patterns of characters within files. You can use the sed and awk online manual pages as a starting place to learn more about these powerful scripting languages.

Scripting Languages

Mac OS X v10.4 Tiger comes with these scripting languages installed:

Language Version Included with Mac OS X v10.4 Website

perl	5.8.6	The Perl Directory
Python	2.3.5	Python Home MacPython Home
php	4.3.10	PHP Home
Ruby	1.8.2	Ruby Home

Table 2: Scripting languages included with Mac OS X

Perl

The Perl language was developed by Larry Wall. An ancient language by computing standards, the first version of Perl was contributed to the alt.comp.sources usenet newsgroup in 1987. Perl was originally developed for text processing, but today some people refer to it as "the duct tape of the internet" because it has proven itself to be so practical as a general purpose language. Perl is often used in web-based programming by executing it as a CGI binary.

There are two ways to run a Perl script. The first is to execute the perl command in a shell terminal, for example

perl myprogram.pl

The second way is to add the following as the first line of the script (this way automatically finds the interpreter in the path, versus the absolute path used for the bash example):

#!/usr/bin/env perl

As with shell scripts, the file containing the Perl script must be made executable with the chmod command.

The three basic data types of Perl are scalars, arrays and key/value pairs, called hashes. Scalars are single values, however, Perl is a bit on the tricky side in that it treats character strings as scalars. An array in Perl is simply a list of values. Perl is tricky here, too. Where most languages limit arrays to hold a single type of value, arrays in Perl can hold any type. A hash is simply a key/value pair. The typical hash maps a string to some value (possibly another string), but you could also map an integer or floating point key to some value. 

As a general purpose language, Perl features all of the usual conditional and looping constructs, such as if-then , while and for loops, and a variant on the for loop called foreach . The foreach loop is used to scan through the elements in a list, such as an array or a hash.

Perl shines the most as a language for processing text using regular expressions, which was its original domain. This topic is far too large for this article, but you can find more information in the extensive Perl Regular Expressions reference documentation.

Python

Python is a general purpose scripting language developed in the early 1990's. It fully supports object-oriented programming, so you can create classes, virtual methods, use inheritance (including multiple inheritance), and operator overloading. 

Python features a large and useful set of built-in data types, including the usual scalars such as integers and floating point values, strings, and collections such as lists, sets, and dictionaries (key/value pairs).

To further aid the development of larger programs, Python allows you divide your work into modules, which are files containing groups of related definitions and statements. You can then import these modules into other modules in order to reuse your code. The Python source file name is the name of the module plus the suffix .py . You can further group modules into packages. 

You can run Python code the same way you run shell and Perl scripts. That is, either by adding the following as the first line of the script:

#!/usr/bin/env python

The second way is by invoking the python command interactively with the name of the file. For example

python myfile.py

The file containing the script must be made executable with the chmod command.

One quirk of Python that takes some getting used to is that indentation is used to perform the function traditionally performed by curly braces ('{ ', and '} ') or begin /end keywords in other languages. That is, blocks of statements are designated by their indentation level, not by virtue of being surrounded in special characters or keywords. Each line in a block of statements must be indented by the same amount of spaces. For example, the following code shows an if construct that executes a block of three statements if the condition is true.

if x < 10:
   print x
   a = a + 1
   print a

Python has an active and growing user community behind it. The Python Tutorial is a great place to start learning more about this dynamic language. If you are considering Python as a general purpose language, you will want to look at the PyObjc project , which provides a bridge between Python and the Objective-C languages.

PHP

PHP is primarily a "server-side" scripting language. PHP code is embedded in a HTML file, to produce dynamic websites and web-based applications. You might surmise that a "server-side" scripting language would require support from the web server, and you are correct. PHP is supported on most major web servers, including Apache, one of the most widely used web servers in the world. Mac OS X v10.4 ships with Apache version 1.3. Even if your web server does not have support for PHP as an integrated module, you can still use PHP as a separate CGI binary. 

PHP can even be used to write command line scripts and even desktop GUI applications such as you've seen with Perl and Python. However, the main use of PHP is in server-side scripting. 

PHP code is embedded in HTML much the same way as Java, on a Java Server Page. PHP code is contained within opening and closing tags, which are recognized by the parser. Everything else outside the tags is ignored. For example, the following HTML contains embedded PHP:

<p>This is an HTML para tag. It will be ignored by PHP</p>
<?php print "Embedded PHP!"; ?>
<p>Another ignored HTML para.</p>

This code uses the PHP print command to output the text "Embedded PHP!" directly into the HTML. You might also surmise that since PHP ignores everything outside of the special tags <?php ?> you can even embed PHP within other kinds of files, such as XML or XHTML files. Again, you are correct.

The basic syntax of PHP draws heavily from C. The basic data types of PHP are: integer, float, bool, and string. The language also has built-in support for arrays and classes. PHP has all the control structures of C, and like Python, blurs the line between scripting languages and general purpose languages even more by supporting constants and an exception handling model.

With any server-side language, security is going to become an issue. With PHP code, you can perform actions which are insecure by definition, such as creating files, executing commands, and so on. But, PHP was designed to be a language used to generate dynamic web content, so you will find a huge range of configuration options to control security. 

The PHP Manual is the best place to start learning more about PHP.

Ruby

Ruby is a scripting language developed by Yukihiro Matsumoto in Japan in the early 1990's. It is a pure object-oriented language, so everything—every string, every number, every user-defined data type—all of these are objects in Ruby. 

Ruby contains the usual data types we've seen so far: integers, floating point numbers, Boolean values, and strings. Again, the interesting thing to note is that even numbers are treated as objects in Ruby, so they respond to messages (methods), like objects you create in C++ or Objective-C. For example, if you have a number in a variable called theNumber , you would find its absolute value by writing:

theNumber.abs

Ruby also supports a data type called a range, which is similar to (but goes far beyond the capability of) the Pascal type. You denote a range of any type (including your own objects) using the '.. ' operator. The following code defines several ranges:

100..1000
'0'..'9'
MyClass.new(2)..MyClass.new(8)

The last line of code defines a range of hypothetical user-written objects of type MyClass . Of course, this flexibility does not come for free. A class that will be used in a range must define the semantics of its comparison and of iteration. 

Ruby provides built-in support for pattern matching through regular expressions. Even a regular expression is an object of type Regexp .

Ruby is gaining popularity as a language to write web-based applications. Ruby scripts can easily be executed with CGI, and Ruby contains a class called CGI that is used specifically for creating CGI scripts. The CGI class has methods for generating HTML and dealing with cookies, among many other capabilities. If you are considering using Ruby to write web-based applications, you won't want to miss the Ruby on Rails project. Rails is an open source web programming framework that is rapidly gaining mindshare due to its simplicity and clean design. 

Integrating Scripting Languages with Xcode Projects

Xcode features a build system that is powerful and flexible enough to handle just about any requirement. You can take advantage of scripting languages during Xcode's build process using:

• External targets
• Run Shell Script build phases
• Shell Script targets 

External targets allow you to run a foreign build system to build a product. The default build tool for an external target is make . You edit the properties of an external target by double-clicking it in the project tree. You may specify the build tool and any arguments it requires, and set the working directory for the build. You may even add your own custom build settings for use by the external build tool. The following image shows the external target property editor in Xcode:

Figure 1: Xcode External Target Property Editor

The Run Shell Script build phase lets you run a shell script using any of the shell languages available on your system. You may add any number of Run Shell Script build phases to a target. You edit the properties of the build phase using the inspector window. You may specify the shell to use to run the script, the script itself, and any input and output files.

If the output of a shell script is used by more than one target in your project, you can add a Shell Script target to aggregate the script or scripts. A Shell Script target contains only Run Shell Script and Copy Files build phases. Any target that uses the Shell Script target's output files can be made to depend on the Shell Script target.

Making Backup Files

The bash example above could be useful to make backup copies of the source files in an Xcode project. Let's look at how the script could be integrated right into the build process.

 The simplest way to integrate the script is to add a Run Shell Script build phase to a target. We'll start the procedure with a project open.

1. Right-click on a target in the project, and choose Add > New Build Phase > New Shell Script Build Phase.

A new build phase with the name Run Script is added as the last build phase for the target.

1. Right-click on the new Run Script build phase and choose Rename.
2. Change the name of the build phase to Backup Files.
3. Open the Inspector Window for the Backup Files build phase.

You will specify the shell to run and the script itself in the Inspector Window. The shell in this case is bash .

1. In the Shell field, enter /bin/bash .
2. Copy and paste the bash script(from the first large code sample in this article), minus the first line, into the Script field. You must remove the first line because the shell field tells Xcode what to run in order to execute the script.
3. Close the Inspector Window

Now build the project, and verify the existence of the bak folder. The folder will contain all the files located in the project directory.

Open Source Repositories and Upgrading

This article only exposes the tip of the programming language iceberg. So far, we have focused on scripting languages, but Mac OS X, with its ties to the world of UNIX, can take advantage of almost any open source project available today. Mac OS X opens up a huge range of tools to work with—from programming languages to applications.

One of the largest, if not the largest repository of open source projects can be found at SourceForge.net .  The SourceForge repository recently surpassed 100,000 Open Source projects. One of the projects hosted at SourceForge, called Fink , is dedicated entirely to porting UNIX open source software to Darwin and Mac OS X.

The DarwinPorts project is similar to Fink, and the two projects have some overlap. DarwinPorts currently contains some 2,500 ports of open source software projects. 

Fink and DarwinPorts are more than open source repositories. They are projects themselves, or metaprojects if you will. One thing you will quickly discover about open source projects is that they often carry their own complex web of interdependencies. Project management systems like Fink and DarwinPorts take care of tracking those dependencies for you. For example, if project A depends on projects B, and C, both B and C will be downloaded as part of downloading project A. Fink uses package management tools ported from Debian GNU/Linux to create its own comprehensive dependency and build tracking system. 

With so many projects available, you need a way to quickly search for the one that best suits your needs. You need to look no further than the Google of open source projects, FreshMeat.net . FreshMeat maintains a huge index of projects for many operating systems, including your favorite and mine, Mac OS X. FreshMeat also hosts general news and information from the open source community at large. It features a mind-boggling array of book reviews, articles, editorials, and much more.

Last but certainly not least, you can look for open source projects on the ADC website. The Apple Open Source Projects (Darwin) page lists projects released under the Apple Public Source License as well as third party projects released under their own licenses.

When You Need to Upgrade

The shells and scripting languages included with Mac OS X are up to date, stable, release versions. However, you might need to upgrade to a newer version, for example, when the maintainer of the language fixes a problem you've been working around. You can install newer versions of any shell or language, however, you should not replace the original version that shipped with Mac OS X. 

How you go about upgrading depends completely on the project and how it is packaged. Fink and DarwinPorts packages will have their own build procedures. Projects that come from other sources vary so widely in packaging and distribution methods it is impossible to cover all the possibilities. Regardless of the source of the software, you will need to study its build and installation scripts carefully to familiarize yourself with its dependencies and target installation folders. 

Many open source developers use GNU autoconf and automake to build and install their software. Autoconf produces scripts that test for the presence or absence of system-specific features. Automake produces the makefile required to build the software on the current system. The advantage of autoconf and automake is that they can adapt the configuration and makefile for many different kinds of UNIX systems. Mac OS X comes with both autoconf and automake installed. You can find out more about autoconf on its online man page, as well on the GNU autoconf website. Automake does not have a man page. Instead, open a shell prompt and type

info automake

or, access the GNU automake website for more information.

Unless you are developing software for the open source community, you probably won't use autoconf and automake directly. Instead, you will use the scripts produced by these programs. 

Some projects use their own configuration tool, like Perl. For an example of building a project from source, let's look at the source distribution of the Perl programming language, available from the CPAN website.

Mac OS X 10.4 ships with Perl 5.8.6, but on CPAN you'll find a minor maintenance update to version 5.8.7. Download the source archive (stable.tar.gz) and uncompress it to a location on your disc.

The Perl source distribution contains a file called README, and platform-specific README files containing further instructions for building Perl on various operating systems. After familiarizing yourself with the project, you will next want to locate the installation instructions. Typical of most open source projects, the Perl source distribution contains a file called INSTALL that tells you how to configure, build, test, and install the software.

Perl is relatively simple to build. The INSTALL file describes the shell commands required to build:

rm -f config.sh Policy.sh
sh Configure -de
make
make test
make install

The first line removes configuration files; they will be regenerated by Perl's Configure script. The make commands build specific targets. The make command without a target builds the Perl executable. The make test target executes many tests to be sure that Perl built and will run properly on your system. Finally, make install copies the executable perl file and any required support files to the default location. In this case, the default location is in /usr/local , so we know this installation will not conflict with the version of Perl shipped with OS X 10.4.

That's it! When the build is finished, you have an updated version of Perl.

Getting Source Code Using CVS

If you are the type who likes to live on the bleeding edge, you will find the world of open source and Mac OS X are ready and willing to accommodate you. Many open source packages let you get the absolute latest and greatest version of their source code, using the Concurrent Versions System, or CVS . CVS is a source control program that tracks changes to individual files. With it, you can check out the most recent version - stable or not, and build it on your local machine. CVS is a command line tool, but there are some graphical front ends for it. Support for CVS, as well as other source control programs, is built into Xcode.

CVS servers usually allow you to check out files anonymously. Most of the projects at the SourceForge repository have good support for anonymous CVS access. Checking out a project on SourceForge is a simple matter of logging in anonymously and executing the CVS "checkout" (co ) command. For example, at a shell command line, you would execute commands similar to the following:

cvs -d:pserver:[e-mail]:/cvsroot/pyxml login
cvs -z3 -d:pserver:[e-mail]:/cvsroot/pyxml co -P xml

The first command logs in anonymously to the PyXML project, which is a Python library for working with XML files. The second line checks out the files in the CVS module xml . When this command completes, the files associated with that CVS project module will be copied to your computer. 

Note: There are many more modules associated with the PyXML project. Projects on SourceForge allow you to browse their CVS repository to discover module names.

Summary

This article has given you a high-level overview of scripting languages, how they are generally used, and how they are shipped with Mac OS X. Hopefully, this article will also encourage you to explore the many open source projects available for Mac OS X. The links below will point you towards additional information on Apple's Developer Center website.

For More Information

• Internet and Web Open Source : For more specific information on installing open source projects on Mac OS X.
• Introduction to Mac OSX Technology Overview: For information on Mac OS X architecture and software development environments. Specifically, see Appendix A, Command Line Primer, for information on executing shell commands.
• Darwin : For information on Apple's own open source project, the Darwin kernel.

Updated: 2005-08-16

<!-- END WIDE COLUMN --><!-- END MAIN CONTENT --><!-- START BOTTOM APPLE NAVIGATION --><!-- googleoff: all --><!-- START FOOTER TABLE -->

Get information on Apple products.
Visit the Apple Store online or at retail locations.
1-800-MY-APPLE

Copyright © 2009 Apple Inc.
All rights reserved. | Terms of use | Privacy Notice

<!-- END FOOTER TABLE --> <!-- googleon: all --> <!-- END BOTTOM APPLE NAVIGATION --> <!-- START CENTER CLOSE --> <!-- END CENTER CLOSE -->