Recently in Perl Category

I've been reading about Google's new DNS service with great interest.  I had to switch to OpenDNS some time ago when my ISP began redirecting 404's to a search page.  This wreaked havoc on my link verification tools.

While reading, I came upon a shell script at http://www.manu-j.com/blog/opendns-alternative-google-dns-rocks/ that I wanted to try.  Since I run WinXP, I translated it to Perl and spruced it up a bit.  I installed the utility from http://members.shaw.ca/nicholas.fong/dig/ and away we went...

# dnstimes.pl - test dns server times

use strict;
use warnings;

my @urls = qw(
	lifehacker.com
	facebook.com
	manu-j.com
	reddit.com
	tb4.fr
	bbc.co.uk
	cindyruppert.com
);

my %dns_servers = (
	Level_3	=> '4.2.2.2',
	Google	=> '8.8.8.8',
	OpenDNS	=> '208.67.222.222',
);

for my $dns_firm (sort keys %dns_servers) {
	my $dns_ip = $dns_servers{$dns_firm};
	for my $url (@urls) {
		my $result = `dig \@$dns_ip $url`;
		my ($time) = $result =~ /Query time: (\d+)/s;
		print "$dns_firm\t$url\t$time\n";
	}
}

Figuring this out was somewhat painful, so I thought I would blog it.

I want to use the same form yaml for several different actions (update, create, view) andI had a list of field names I wanted to remove that were not relevant for create.

The remove_element method only works on the immediate children of the invoking object (as does get element), so $form->remove_element does not usually work.  I found a posting on the HTML::FormFu list where Carl Franks showed the idiom $element->parent->remove_element($element);

So my code became:

# remove fields not needed for this action
for (qw(created_by created_time changed_by changed_time version)) {
	my $element = $form->get_all_element({name => $_});
	if ($element) {
		$element->parent->remove_element($element);
	}
}


Works great!

One of my sites was coming up with a graybar, so I wanted to check it against other hosts, besides the default.  I tried a couple of the websites that I normally do this with, and they weren't working.  So, I wrote the following few lines.

use strict;
use warnings;
 
use WWW::Google::PageRank;

my @hosts = (
	"toolbarqueries.google.com",
	"209.85.227.147",
	"209.85.227.104",
);

for my $host (@hosts) {
	print "\n$host\n";
	my $pr = WWW::Google::PageRank->new(host => $host)
		or die $!;
	for my $domain (@ARGV) {
		$domain =~ s{http://}{};
		$domain =~ s{/$}{};
		my $pagerank = $pr->get("http://$domain/");
		print "    $pagerank  $domain\n";
	}
}

Here is a script that deletes old files.  In the Unix environment this is an easy job for a one line script using find.  This Perl version has the -test option, to allow you to see what will happen, and prints some totals.

#!/usr/bin/perl

# delold.pl - delete old files

#-----------------------------------------------------------
# 05/24/2009  WR  Written
#-----------------------------------------------------------

use strict;
use warnings;
use Win32::Autoglob;
use Getopt::Long;

$| = 1;

sub print_usage;
sub abort_usage;

# option defaults
my $days_back    = 10;
my $test_flag    = 0;
my $verbose_flag = 0;
my $quiet_flag   = 0;

# get options
GetOptions (
    'days=i'  => \$days_back,
    'test'    => \$test_flag,
    'verbose' => \$verbose_flag,
    'quiet'   => \$quiet_flag,
    'usage'   => sub {print_usage; exit 1},
) or abort_usage "Invalid option";

my $time = time();

my $files_to_delete = 0;
my $files_total     = 0;
my $files_failed    = 0;

FILE:
for my $file (@ARGV) {

    # must exist, must be a plain old file
    next FILE if !-e $file;
    next FILE if !-f $file;

    $files_total++;

    # modified age in days
    my ($mtime) = (stat($file))[9];
    my $modified_age = ($time - $mtime) / (3600 * 24);

    # skip if too young
    next FILE if $modified_age < $days_back;
    
    $files_to_delete++;

    if ($test_flag || $verbose_flag) {
        printf "Age: %-6.1f  File: %s\n",
            $modified_age,
            $file;
    }

    # skip if we are testing
    next FILE if $test_flag;

    # delete the file
    if (!unlink $file) {
        warn "Failed to delete file $file\n";
        $files_failed++;
    }

}

if ($test_flag) {
    print "\nTest Flag is set, no deletes done!\n";
}

my $files_remaining = $files_total - $files_to_delete + $files_failed;

if (!$quiet_flag || $test_flag) {
    print "\n";
    print "Total files:      $files_total\n";
    print "Files to delete:  $files_to_delete\n";
    print "Failed to delete: $files_failed\n";
    print "Files remaining:  $files_remaining\n";
}

exit 1;

sub abort_usage {
    print STDERR join("\n", @_), "\n" if @_;
    print_usage;
    exit 0;
}

sub print_usage {
    print STDERR <<END;
Usage: delold.pl [Options] files...
Options:
    --days n      - Set age of files to keep.
                    Files over "days" old will be deleted.
                    Default is 10 days.
    --test        - Print file names to be deleted with age,
                    but do not actually delete.
                    Default is false.
    --verbose     - Print file name and age while deleting.
                    Default is false.
    --quiet       - Suppress printing of totals after deleting.
                    Default is false.
    --usage       - print this message and exit
END
}

The Perl community is making an effort to improve visibility.  I added "Perl programming" to the top of the blog in support of this effort.

Here is a quick script I wrote to load SQLite tables from CSV files.  I exported the files from Microsoft Access.  The first row has to have correct column names.

Text::xSV is the best CSV file handler I could find on CPAN.  It correctly handles files with embedded newlines as well as very large files.  SQL::Abstract makes it very easy to use straight DBI to speed up the load without going through an ORM. 

I prepare each row separately because I may have cases where there are trailing unused fields, and some Microsoft programs (Outlook comes to mind) do not attempt to output empty fields for them.  If the speed difference is important, you may want to try to guarantee the exact number of fields in each row so you can prepare once.

#!/usr/bin/perl

# load_csv.pl - Load a db table from a csv file.
#               First row must be field names.
# 04/28/2009  W. Ruppert

use strict;
use warnings;
use Text::xSV;
use DBI;
use SQL::Abstract;

$| = 1;

sub usage {
    my ($msg) = @_;
    warn "$msg\n" if $msg;
    die "Usage: load_csv.pl db table csvfile\n";
}

# get parms
my $db        = shift || usage "No database name";
my $table     = shift || usage "No table name";
my $data_file = shift || usage "No csv data file";

usage "No such file: $db"        unless -e $db;
usage "No such file: $data_file" unless -e $data_file;

# setup csv file
my $csv = new Text::xSV;
$csv->open_file("$data_file");
$csv->read_header();

my $sql = SQL::Abstract->new;

# connect to db
my $dbh = DBI->connect("dbi:SQLite:$db", "","",
        { RaiseError => 1, PrintError => 0 }
) or die "can't connect\n";

$dbh->do('begin');

my $max_commit  = 1000;
my $inserted    = 0;

# process csv rows
while (my %fieldvals = $csv->fetchrow_hash) {

    # SQL::Abstract sets up the DBI variables
    my($stmt, @bind) = $sql->insert($table, \%fieldvals);

    # insert the row
    my $sth = $dbh->prepare($stmt);
    $inserted += $sth->execute(@bind);

    # progress bar
    print "*"            unless $inserted % 10;
    print " $inserted\n" unless $inserted % 500;

    # commit every once in a while
    unless ($inserted % $max_commit) {
        $dbh->do('commit');
        $dbh->do('begin');
    }
}

$dbh->do('commit');
$dbh->disconnect;

print "\nInserted $inserted records\n";
exit 1;

Need to get the size of an image on the web without storing it.  Done by pulling the image with LWP::Simple and giving the content buffer to Image::Size.

#!/usr/bin/perl
# Get dimensions of web image
# 04/17/2009  WR

use strict;
use warnings;
use LWP::Simple;
use Image::Size;

my @urls = (
	"http://www.google.com/intl/en_ALL/images/logo.gif",
	"http://l.yimg.com/a/i/ww/beta/y3.gif",
	"http://www.example.com/nothing.gif",
	"http://graphics8.nytimes.com/images/misc/nytlogo379x64.gif",
);

URL:
for my $url (@urls) {
	my $image = get $url;
	unless (defined $image) {
		warn "Couldn't get $url!\n";
		next URL;
	}
	my ($width, $height) = imgsize(\$image);
	printf "%4d %4d   %s\n", $width, $height, $url;
}

exit 1;