At first glance, it would seem that you are out of luck if what you want to automate requires human intervention. There are things that need someone to pick from menu options, enter passwords, or make decisions based on the information presented. Interactive applications require a user’s reaction, don’t they? The answer for the cleverly lazy system administrator is “Not always” thanks to a little program called expect.
While I had heard of expect sometime before, I discovered a few years ago just how useful this language is. My partner and I were developing a Web-based system that required regular updates from the main computer’s database, a database that would not allow command-line scripting. The data we needed required the execution of an SQL statement that could only be entered through the vendor’s menu interface. That SQL statement would then generate the data file we needed for the Web interface. The whole process hinged on writing something that mimicked a user sitting at a terminal entering information as the various prompts were presented to him or her. Expect, a software suite/language based on Tcl, was the answer to our dilemma. Later, Expect would make it possible to stretch our Web-tool well beyond what we, ahem, expected at the time.
This article is, for the most part, an excerpt from my first book, “Linux System Administration: A User’s Guide”, published in 2001.
— Marcel Gagné
Still wondering if you need it? Remember that laziness discussion at the beginning of the chapter? Well, pretend that you are working late and that the last thing you need to do before leaving is to log onto your remote site, make sure that a specific application has completed (it is always done by 3:00 AM), and then download the file that application generates back to your local site. It is now 10:00 PM and you would much rather go home than wait for the magic moment when the file is ready. You could just launch an at job that starts ncftp for the download, but you don’t know the filename because the output name changes at each run. You find the name by logging into the menu system and checking the completion log. (I am purposely making this complicated to demonstrate that there are instances that are hard to automate with a simple shell script.)
The basic format of an expect script is this:
#!/usr/local/bin/expect # Comments on this script (name, what it does, optional) spawn some_command set response myanswer expect "Some prompt . . . . " send $response\r close
Here’s what happens. The spawn keyword tells expect to begin some program. This could be a shell (spawn /bin/bash) or any kind of command through which the session will take place. With the set keyword, you set the variable response to some predetermined response. The language’s namesake, the expect keyword, does exactly what it sounds like it does. It scans the output of whatever command you invoked with spawn, searching for matching text. Then, send responds to the expected text with the first variable, $response. Let’s do something real now.
I run an Apache Web server on my system with OpenSSL extensions for secure transactions. Starting Apache with the OpenSSL extensions running requires me to enter a security pass phrase in order for it to start up because the private key files on the server are encrypted. This is all fine if I am there to enter the pass phrase, but what happens if the server goes down when I am not there? It hasn’t happened for months, but these things can happen and I do go on vacation sometimes. It could be something as crazy as me adding a SCSI card for my new tape drive. I might have forgotten (it has happened) to restart the Web server with OpenSSL running. What then? To get around this problem, I wrote the following simple expect script:
#!/usr/bin/expect # Routine: startapachessl # Purpose: Start Web server with OpenSSL active # log_file -a /tmp/expectlog #log_user 0 spawn /bin/bash sleep .2 send "/usr/local/apache/bin/apachectl startssl\r" expect "Enter pass phrase*" sleep .2 send "mysecretphrasegoeshere\r" sleep .2 close
When the system restarts, whether I am there or not, this script will restart my Apache Web server with OpenSSL running. Looking at the script, you’ll notice a couple of interesting things. For instance, the log_file parameter is new. What this does, as you might expect, is define a log file for the execution of this script. Whether the file is written to or not is defined by the log_user parameter. If the parameter is set to 1, logging will take place. I tend to use log_user when I am still testing the script, but you may decide you want to capture the output at all times. Notice as well that I am spawning a bash shell to execute the script that starts my server. Then, there are the sleep statements. In all cases, I have the shell wait 1⁄5 second before continuing on. Finally, the close statement tells the spawned process that there is nothing more to come. At this time, expect terminates and returns to the process that spawned it.
There is no doubt that you could program these functions with other languages, but expect makes it easy. What you will find as you go along is that not every tool is perfect for every job. For quick and dirty automation of interactive applications, nothing beats expect.
Fully exploring expect would require a book on its own (in fact, there is one) and what I am trying to do is give you a taste of what you can do with it rather than explaining every aspect of the language. Before I let you run off to do your own exploring, let me take these examples one step further.
We all know that changing passwords on a regular basis is a good thing, as is choosing good passwords, and it is a fairly easy thing to have a user do when he or she logs in but somewhat more difficult if the user does not have a login account. I’m talking about e-mail-only users, the ones who you allow POP3 mail pickup (or Web-based e-mail) but no actual command prompt access. A number of offices have precisely this kind of setup for their Linux systems—it serves as an e-mail or Internet gateway and allows no logins. So, how do you allow users who aren’t allowed to login to change their passwords? After all, changing passwords is an interactive activity as the following dialogue will attest.
[root@myhost] # passwd New UNIX password:
Even more complicated is that in order for a nonroot user to change his or her password, that user must first enter his or her old password, so the dialogue is even more complex. What now?
You could create a Web-based form whereby a user could enter all that information up front (see Figure 1).
Figure 1 Modifying passwords through a Web browser
A Perl script behind the form would extract the variables and pass them to an expect script that does the rest. If you are curious about this little Web application or would like to use it, feel free to download it from my Web site (http://www.marcelgagne.com/downloads.html).
In the meantime, have a look at this segment from the application:
#!/usr/bin/expect
# Routine: psdcmd
# Purpose: to change a user's password with expect
log_user 1
set uservar [lindex $argv 0]
set currpassword [lindex $argv 1]
set newpassword [lindex $argv 2]
set renewpassword [lindex $argv 3]
#
# log_file -a /tmp/expectlog
# send_user "Spawning passwd command with uservar.\n"
spawn su -l -c "passwd" $uservar
expect "Password:"
sleep .1
send "$currpassword\r"
sleep .1
#
expect {
"(current) UNIX password:" {send "$currpassword\r"}
"su: incorrect password" {exit 0}
}
sleep .1
expect {
"su: incorrect password" {exit 0}
"New UNIX password:" {send "$newpassword\r"}
}
sleep .1
expect {
"BAD PASSWORD:" {exit 0}
"Retype new UNIX password:" {send "$renewpassword\r"}
}
sleep .1
expect {
"su: incorrect password" {exit 0}
"New UNIX password:" {exit 0}
}
#End of password change routine
The set varname [lindex $argv num] construct represents arguments passed to the expect routine. Notice that the spawn parameter call does an su to the user name in order to change the password. By default, CGI scripts on a Web server execute as some unprivileged user like nobody or www, so you need to change your effective user in order to change the password. Those names aren’t etched in stone, by the way. Other servers use www-data or apache. Check with your distribution to make sure.
There is one other new item in the script. Look at the send_user parameter. This is essentially a print statement. I left it in the sample script because I wanted to show you a clever way of debugging your expect scripts. Every programmer has inserted debug statements into his or her code to monitor how things were going. This is the same idea in this case. You can use send_user as a means of communicating with the outside world in the course of the script’s execution. Because I capture the output of the expect script through my Perl script, I will see these messages as well.
By the way, the Perl script calls the routine in this way (the entire command is on one line):
$return_code = `./psdcmd "$username" "$currpassword" "$newpassword" "$renewpassword" `;
As you can see, the expect script is called with the user name, current password, the new password, and the new password repeated. I could simply have passed the new password twice, but I wanted to keep the verification aspect of the password change routine as close to what the user would experience as possible at the command line. More to the point, it is probably also a good idea to force the user to confirm the password before changing things on him or her.
Automating Interactive Automation
Now that you have had your introduction to scripting with expect, I am going to make the process almost impossibly easy. Rather than manually creating an expect script, how about letting a program do that for you, too? When you install expect, you will also install a little program called autoexpect. Simply put, autoexpect watches whatever you are doing in an interactive session and creates the expect script for you. Here is the format of the command:
autoexpect -f script_outputfile command_string
For instance, imagine that you want to log into a remote system that is behind a firewall—essentially a two-step login process. After you log into the firewall, you execute a login (telnet, ssh, and so on) to yet another system on the internal network, and then you execute a standard menu program. What you would like is to have this whole process of logging in twice and starting the menu automated for you. From the command prompt, type this command:
autoexpect -f superlogin.script telnet firewall.mycompany.com
When you have finished your login, you can exit the menu and log out. Just like magic, autoexpect will have captured the entire session for you. Now, you will probably want to do some editing in there, but the basics of the script and all the prompts are captured there for you. Make the script executable and you are almost done. There is still one other thing you will want to add. At the end of your new expect script, add this command:
interact
This tells expect to return control to you after it has done its work. Without it, expect closes the spawned process and all you’ve managed to do is log in and log out very quickly.
In no way do I intend this to be the definitive reference on expect. I do, however, hope that this little introduction (indeed, this whole chapter) will serve to whet your appetite and inspire you to explore other ways of developing constructive laziness. After all, we all have other work to do.
What’s all this on your screen about a magic cloak?
This article is, for the most part, an excerpt from my first book, “Linux System Administration: A User’s Guide”, published in 2001.
— Marcel Gagné