I’ve ported my additional DTrace probes to blead perl, adding the following probes

• new-sv and del-sv to track Perl allocations,
• main-enter and main-exit to show what part of Perl’s execution phase we’re in
• load-module-entry and load-module-return to instrument use, require, do.

I’ve also started to document the probes at http://wikis.sun.com/display/DTrace/perl+Provider
To use the patch,

1. get bleadperl:
   1. rsync -avz –exclude .svn/ –delete rsync://ftp.linux.activestate.com/perl-current/ bleadperl
2. apply the patch:
3. cd bleadperl ; patch ../bleadperl.diff
4. run configure with dtrace enabled:
   1. ./Configure -de -Dusedevel -Dinc_version_list=none -Dprefix=/usr/local/bleadperl/ -Dldflags=-Dman3ext=3pm -Duseithreads -Duseshrplib -Uversiononly -Dusedtrace -Doptimize=-g
5. make
6. make test
7. make install
8. run./perl.d (also found at http://distributedinformation.com/DTrace/ ) and then run whatever Perl code you want to instrument.

Given the following functionally similar Perl modules, I was curious what the internal Perl differences in execution were.

To recap – (from PerlDoc)

• Perl use – Imports some semantics into the current package from the named module – It is exactly equivalent to

      BEGIN { require Module; Module->import( LIST ); }

• Perl require – demands that a library file be included if it hasn’t already been included. There is pseudocode there to suggest that a previously required module (successful or not) would just return the previous result

So given a set of modules that don’t define an import() function, one would expect that there would be little difference.

The (1/0) tuples in braces are counts of allocations and deallocations by the Perl interpreter in in that enter-return pair, and execution times are all using DTrace’s vtimestamp, so are adjusted for time on CPU.

This suggests to me that Perl use and require suffer from the same problems that are encountered when using #include’s inside header files in large scale C and C++, a massive, un-realized parser mess, due to the re-importation of dependencies for the short term convenience of the developer.

While the require output appears better, none the less (looking at the source code for Perl_ck_require) it does more than a nop the second time around (at minimum, its doing a single allocation, and converting the Module name to a File.pm). It is interesting (scary really) that the use case has many more allocations than the require case, but I presume that has to do with the sillyness of importing code into the same namespace several times. That main.pl’s BEGIN is parsed&allocated for 9 times, and Module2:BEGIN is parsed&allocated for twice – is not really what I had in mind.

For those of us working with large scale Perl code (in my case TWiki and Catalyst) it seems that there is not enough clarity, and there are opposing concerns for users that run Perl directly, incurring the parse and allocate steps each time, and those using accelerators such as mod_perl and speedy_cgi, who essentially start at the main-run phase (I think).

Reducing the above examples by removing the use or requires from everywhere except the top level main.pl, still shows an unexpected side effect that the use case has 34 more allocations and 26 deallocations – but… there is less going on. So it is possible, that the inconvenience of needing to manage the Perl module loading at the top level might be worth while, at least until Perl is changed to do what the ‘manual’ implies.

Another interesting side effect of replacing use with require, is that it moves the execution of the BEGIN from the parse phase to the execution phase – again reducing the effectiveness of Perl accelerators, but perhaps more inline with simplistic expectations.

Seems to me its time to move to bleadperl, and see if I can offer a fix.

The above tests are done using the dtrace enabled Perl 5.8.8 that is in my svn repository.

With the load-module probe, you can see at what point modules are loaded using ‘do’, ‘use’ or ‘require’.

The following code:

#!/usr/local/bin/perl -w

use strict;

my $initial = "there once was a fish. Its feet were small";
my $post = func($initial);
my $post2 = func($initial);
print "$post\n";

do 'call2.pl';
#eval `cat call2.pl`;
#use CGI::Session;

sub func {
    $_[0] =~ s/there/There/;
    return $_[0];
}

produces the following output:

 == call1.pl ==========================================================
  perl*::perl_alloc:main-enter
  perl*::perl_alloc:main-exit,  (0/0) (56 uS)
  perl*::perl_construct:main-enter
  perl*::perl_construct:main-exit,  (12/0) (624 uS)
  perl*::perl_parse:main-enter
>>>>>>>>>> perl*::Perl_utilize:load-module-start (strict)
>>>>>>>>>> perl*::Perl_ck_require:load-module-start (strict)
<<<<<<<<<< perl*::Perl_ck_require:load-module-end (strict.pm) (3 uS)
>>>>>>>>>> perl*::Perl_ck_require:load-module-start (Carp)
<<<<<<<<<< perl*::Perl_ck_require:load-module-end (Carp.pm) (2 uS)
>>>>>>>>>> perl*::Perl_dofile:load-module-start (call2.pl)
<<<<<<<<<< perl*::Perl_dofile:load-module-end (call2.pl) (3 uS)
  perl*::perl_parse:main-exit,  (299/46) (3069 uS)
  perl*::perl_run:main-enter
>>>>>>>>>> perl*::Perl_utilize:load-module-start (strict)
>>>>>>>>>> perl*::Perl_ck_require:load-module-start (strict)
<<<<<<<<<< perl*::Perl_ck_require:load-module-end (strict.pm) (3 uS)
  perl*::perl_run:main-exit,  (69/45) (533 uS)
  perl*::perl_destruct:main-enter
  perl*::perl_destruct:main-exit,  (0/3) (24 uS)

see my perl5.8 trac for the code -or grab the modified 5.8.8 source from svn

the  perl*::Perl_utilize:load-module-end probe is currently being worked on, and I really would like to see the actual parse component of the load separated out.

I’ve just created a subversion repository with perl 5.8.8, and the accumulated DTrace patches – including the using is-enabled to reduce the performance impact of the Probes when disabled. Byran and I, (and anyone else that would like to help) will be working slowly towards making Perl a first class DTrace citizen over the coming months. Next stop – Perl Guts Illustrated

Of course, we’ll also port it all to Perl 5.10 – the 20th anniversary release

I’ve been promising myself some time to try out DTrace on TWiki’s codebase for over a year. By following Bryan Allen’s
instructions using Richard Dawe’s adaption of Alan Burlison’s work… I now have a Perl 5.8.8 with DTrace probes.

Sounds great, except for one thing…. I now have to learn enough about DTrace to use it The patch that Alan and Richard have (or at least their DTrace scripts) seem to require a priori knowledge of the Perl process’ pid… not something thats going to work out for what I want to do.

For a quick test, DTrace -c ./view -s /export/home/sven/src/dtrace/subs-tree.d does show the program flow.

The following is while running some perl scripts – the 2 numbers are their pids.

# dtrace -l | grep -i perl

17803  perl17669        libperl.so                      Perl_pp_sort sub-entry

17804  perl17669        libperl.so                   Perl_pp_dbstate sub-entry

17805  perl17669        libperl.so                  Perl_pp_entersub sub-entry

17806  perl17669        libperl.so                      Perl_pp_last sub-return

17807  perl17669        libperl.so                    Perl_pp_return sub-return

17808  perl17669        libperl.so                     Perl_dounwind sub-return

17809  perl17669        libperl.so                Perl_pp_leavesublv sub-return

17810  perl17669        libperl.so                  Perl_pp_leavesub sub-return

88501  perl17760        libperl.so                      Perl_pp_sort sub-entry

88502  perl17760        libperl.so                   Perl_pp_dbstate sub-entry

88503  perl17760        libperl.so                  Perl_pp_entersub sub-entry

88504  perl17760        libperl.so                      Perl_pp_last sub-return

88505  perl17760        libperl.so                    Perl_pp_return sub-return

88506  perl17760        libperl.so                     Perl_dounwind sub-return

88507  perl17760        libperl.so                Perl_pp_leavesublv sub-return

88508  perl17760        libperl.so                  Perl_pp_leavesub sub-return

so… first ignorant modification – in subs-tree.d, it wants to trace perl$target:::sub-entry – change that to perl*:::sub-entry, and of course, it works exactly as I want – attaches to all subsequent perl process (running my dtrace-perl build) and tells me whats going on. The only caveat being that the DTrace script will only start if there is a Perl process running – the provider is obviously not persistent.

Brilliant!

Should be a fun Christmas holiday adventure – 410 pages of dtrace book, and a myriad of web pages to consume and digest.