Scripting macOS, part 4: Running the Script

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

• Part 1: First Script
• Part 2: The Script File
• Part 3: The Code
• Part 4: Running the Script
• Part 5: Lists of Commands
• Part 6: Turning it off and on again
• Part 7: Download and Install Firefox

I will publish one part every week over the summer. Enjoy!

Running the Script

Now that we have the code for a functional (even though minimal) script, you can run it from the command line with:

> ./hello.sh

(Verify that the current directory is your script project folder.) You probably have wondered why you need the ./ before the script name. When you type the script name without the ./ it will fail:

> hello.sh
zsh: command not found: hello.sh

What does the ./ mean and why do we need it?

Current Working Directory

In the shell, the . character represents the current working directory or the directory that you ‘cd-ed’ to. Usually, the working directory is shown in the window title bar of Terminal and in the shell prompt (the status text shown before your cursor in the Terminal window). You can also have the system show your current working directory with the pwd command:

> pwd
/Users/armin/Projects/ScriptingMacOS

When you type ./hello.sh the shell will substitute the current working directory for the . character to get

/Users/armin/Projects/ScriptingMacOS/hello.sh

Your path will of course look different.

This means that when I use the ./hello.sh form, the shell knows exactly which file I want to execute and where to look for it. There is no ambiguity.

Now we know what the ./ means, but why is it necessary?

Finding Commands

A command you enter in the shell is just a string of text to begin with. The shell has to parse this text into pieces to determine what needs to be done.
We will use the chmod command from earlier as an example:

> chmod +x hello.sh

The text you entered is ‘chmod +x hello.sh’. The shell will split this text into pieces on the spaces (or tab characters). This will yield three elements:
‘chmod,’ ‘+x,’ and ‘hello.sh’

The shell is only really interested in the first element. The shell will try to interpret the first element of your entry as a command.

Note: Many programming languages start counting at zero. This first element is also called ‘argument zero’ or $0.

The remaining elements are arguments, which the shell will pass on to the command. Arguments are optional. Not all commands require or even have arguments.

Note: This is a simplified description of the parsing process in shells. The reality is quite a bit more complex. But this description is ‘close enough’ for most situations. We will explore some of the nuances later.

If you are interested, you can get all the details in the shell’s documentation:

• Zsh Documentation: Shell Grammar
• bash man page, search for ‘Command Execution’

When there is a ‘/‘ character anywhere in the first element, the shell will interpret the first element as a full or relative path to an executable file and attempt to run that.

This is the behavior we are using when we type ./hello.sh. Since that contains a ‘/‘ the shell will resolve the path and execute our script.

We are intentionally using the seemingly redundant ./ prefix to tell the shell to run the script from the current working directory.

When the first element does not contain a ‘/,’ the shell will check if it is one of the following:

• a shell function
• a shell built-in command or reserved word
• an external command

Shell Functions

Shell functions are (mainly) customized shell behavior declared by the user. They look and work like commands.

Note: If you are interested in customizing your shell environment, you can find details in my books “Moving to zsh” and “macOS Terminal and Shell.”

Shell Built-ins

Shell built-in commands cover tasks that are either inherent to the shell or can be performed much faster within the shell than as an external command. These are commands that affect the internal state of the current shell process like cd, alias, or history, or commands that are simpler and faster to implement as built-ins like echo or read.

You rarely need to worry about whether a command is a built-in.

Many built-in commands will have an executable external command file as well. This serves as a ‘fallback’ for the rare situation that the shell built-in is not available.

You can use the (aptly, but confusingly, named) command built-in to determine if a command is built-in or external:

> command -V cd
cd is a shell builtin
> command -V sw_vers
sw_vers is /usr/bin/sw_vers

External Commands

When the command entered is neither a function, nor a built-in, and does not contain a ‘/,’ then the shell will go search for an executable file with that name.

The PATH environment variable determines the locations in the file system and the order in which to search them.

The default PATH on macOS in an interactive shell is:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The PATH variable contains a colon-separated list of directories. When you have third-party software installed, or customized your shell configuration, your shell may have additional directories. The default PATH splits into the following five directories:

/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin

Note: Variable names in the shell are case-sensitive. The variable names MY_VAR and my_var represent different variable and values.
Zsh, however, has the concept of ‘connected variables.’ In zsh, PATH and path are connected variables. The upper-case PATH contains the colon-separated list of directory paths, while the lower-case path is an array of paths. The actual list of directories will be the same and changing one variable will change the other. Zsh has this concept to maintain compatibility with other shells with the PATH, while also allowing you to use array operators on the path.
I will use the colon-separated PATH, because it is more compatible with other shells.

When a user enters a command like this:

> system_profiler

This is neither a function or alias, nor a built-in command. It also does not contain a ‘/.’

The shell will check for the presence of an executable file with the name system_profiler in all the directories given in the PATH. It will start with /usr/local/bin, then /usr/bin, then /bin, until it finally finds a matching file in /usr/sbin.

Then the shell will attempt to execute /usr/sbin/system_profiler.

When no matching files can be found in any of the directories listed in the PATH, the shell will present a ‘command not found’ error.

> cantFindMe
zsh: command not found: cantFindMe

If you are curious which file the shell will use for a given command you can use the command or the which tool:

> command -V system_profiler
system_profiler is /usr/sbin/system_profiler
> which system_profiler
/usr/sbin/system_profiler

PATH precedence

Once the shell finds a matching file, it will stop searching the remaining paths. If there were a second matching executable in /sbin, the shell would never find and execute it. The order of the directory paths given in the PATH variable determines the precedence.

We can test this by placing an executable with a file name matching an existing command in /usr/local/bin. Since /usr/local/bin comes first in the default interactive PATH, the shell should prefer our executable over the default command.

> sudo ditto hello.sh /usr/local/bin/system_profiler

You need administrator privileges to modify the contents of /usr/local/bin. The ditto command preserves all the file’s metadata (like privileges and extended attributes), which makes it preferable to cp for this task.

Then open a new Terminal window. You need to do this because every shell instance will cache or ‘remember’ the lookup for a command to speed up the process later. The shell instance in your current terminal window will remember the last lookup for the system_profiler command. A new Terminal window will start a new ‘fresh’ shell instance, forcing a new lookup:

> which system_profiler
/usr/local/bin/system_profiler
> system_profiler
Hello, World!

Obviously, overriding a system provided command this way can break your workflows and scripts. To be safe, remove our script from /usr/local/bin right away:

> sudo rm /usr/local/bin/system_profiler

There are some situations where overriding a system-provided command is desirable, though. For example, you could install the latest version of bash 5 as /usr/local/bin/bash. And then, when you invoke bash from your interactive terminal, you will launch that version, instead of the outdated version that comes with macOS.

Note: You cannot simply overwrite /bin/bash with the newer version on macOS. The /usr/bin, /bin, /usr/sbin, and /sbin folders are protected by System Integrity Protection and the read-only system volume.

When bash is invoked with the absolute path /bin/bash, the path will need to be updated to use the newer version, though. This includes the shebangs in scripts, and the UserShell attribute in a user’s account record for their default shell.

Extending your tool set

As you get more confident and experienced with scripting, you will assemble a set of scripts that you will use regularly. When you use a script often, it would be nice if the shell recognized them as commands, without having to type the path to them.

To achieve that you can add the directory containing the scripts to the PATH variable. I put my frequently used tools in ~/bin. The name is chosen to be somewhat consistent with the four standard locations for tools, but really does not matter. You can append your tool directory to the PATH with:

> export PATH=$PATH:~/bin

This reads as: replace the value of the environment variable PATH with its current value and append :~/bin. You can verify the new value with

> echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/armin/bin

We added our custom directory to the end of the PATH variable, so you cannot accidentally override system tools.

To prevent other local users from changing your command files, you should set the directory’s privileges, so that only you can access:

> chmod 700 ~/bin

Now you can copy our hello.sh script to that folder. We will rename it to just hello, so it looks more like a command:

> ditto hello.sh ~/bin/hello

Then you can run your script just by typing

> hello
Hello, World!

Most of the scripts you build will not need to be as accessible as this.

Since you do not want to manually change the PATH every time you create a new Terminal window, you should add this line to your shell configuration file:

export PATH=$PATH:~/bin

Context Matters

Leveraging PATH to find and possibly override commands can be a useful and powerful tool. However, when scripting, you have to always remember, that the interactive Terminal environment may not be the environment that your script will run in ‘production.’

This is especially important for scripts run by

• LaunchDaemons or LaunchAgents
• AppleScripts
• applications other than Terminal (such as Xcode)
• installer packages
• management systems

When run from any of these environments, the environment the script runs in will be different from your interactive shell environment. Most likely the PATH in any of these environments will be set to the minimal four system folders:

/usr/bin:/bin:/usr/sbin:/sbin

This should work well for most commands and tools. But when you are working with third party software, especially newer versions, then you need to pay very close attention.

When you build scripts for these environments, I recommend setting the PATH at the beginning of your script explicitly to the value you need:

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

When your scripts require tools, include their locations in the PATH. This way, the PATH environment is declared explicitly at the beginning of the script and there can be no confusion.

For the scripts in this series, the default PATH will be sufficient, so we will not need to worry about this.

Next: Lists of Commands