fosiki: collaboration systems development, taking TWiki further

Jul 25 2008

Debian TWiki repository now with 212 TWiki Plugins, Contribs, Skins and more.

Tag: debian, enterprise, environment, new, twiki, twikiapplication, wiki — Sven Dowideit @ 3:09 pm

I’ve just updated the Experimental TWiki and Plugins repository. It now contains TWiki 4.2.0 and 212 Plugins, Contribs and Skins that you can simply apt-get install

To use them, add the following 2 lines to your /etc/apt/sources.list
deb http://distributedinformation.com/experimental/ experimental main contrib deb-src http://distributedinformation.com/experimental/ experimental main contrib

then type
apt-get update
to update the available packages.

you can now see all 212 packages with apt-cache search twiki-

and install (assuming you don’t have twiki installed yet)

apt-get install apache2 twiki

and TWiki Contrib installation is as easy as

apt-get install twiki-bugscontrib

You will still need to use configure to enable Plugins.

Please report your experiences to me - bugs, gripes, you name it - its a work in progress. and I need your help!

Comments (0)

Jun 29 2008

defense against the dark arts? (Cross site scripting and Cross Site forgery)

Tag: debian, enterprise, twiki, twikiapplication — Sven Dowideit @ 12:12 am

I was having a discussion with someone on IRC about how TWiki is vulnerable to Cross-site scripting and Cross-site request forgery, and we realized that there are 2 possible approaches to securing TWiki effectively (both requiring a unique magic number for all URLs):

add a pre process to the TWiki::UI system, requiring a valid and unique magic, and a post process step between rendering and output to the browser
use a small proxy system between TWiki and browsers to add and validate the magic

1. is actually still risky as all scripts still are able to output directly to the browser using a =print= statement, thus giving the user urls that may not have a necessary magic in the url, or similarly for AddOns that persist in not using resthandlers.

whereas 2. abstracts the security from the application server, in much the same way as it is for ssl - goodness all round.

So - I wonder if there is such a proxy already?

There are also massive performance reasons why you should always have a proxy between browsers and heavy application servers like TWiki - this too could do with filling out.Securing TWiki is not as simple as converting all actions to POST (ie using proper REST / HTTP) because there are too many legacy conveniences, allowing GET URL’s to act upon the data. But, by delegating the securing of the transactions to an external wrapper, I think we can avoid these flaws.
see Wikipedia on Cross Site Scripting and Cross-site request forgery

Comments (0)

May 02 2008

Firefox 3 pre release builds includes 64bit

Tag: debian, new — Sven Dowideit @ 3:17 pm

I’ve been using firefox 3 on my notebook since beta3, and loving its lower CPU and memory needs, but have been frustrated by the lack of 64 bit builds for my 8Gig RAM desktop development and VMWare system.

It seems that they have been building 64bit binaries for quite some time - see their nightly build dir.. http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/

NICE!

Comments (0)

Apr 15 2008

progress towards TWiki 4.2.1 patch release.

Tag: new, perl, twiki, twikiapplication, wiki — Sven Dowideit @ 3:57 pm

(test post using Gnome Webog poster)

The last fortnight, I’ve been working on creating a new, soon to be released to web 2.0 ‘Beta’ site based on TWiki 4.2.0. That means that I’ve been busy finding, reporting and fixing Bugs for TWiki 4.2.1.

The list todate are:

Item5455 BuildContrib doesn’t cope with larger numbers of files Closed 15 Apr 2008 - 04:34 SvenDowideit

Item5536 robots.txt is missing some obvious scripts - like login Waiting for Release 14 Apr 2008 - 08:43 SvenDowideit

Item5535 IF{”$BANNER” does not really work. Waiting for Release 14 Apr 2008 - 06:49 SvenDowideit

Item5534 missing contexts for several bin scripts Waiting for Release 14 Apr 2008 - 06:25 SvenDowideit

Item5533 tmpl login script does not do writeCompletePage, so it does not get addToHEAD bits. Waiting for Release 14 Apr 2008 - 05:10 SvenDowideit

Item5513 update TalkContrib topic Closed 11 Apr 2008 - 06:19 SvenDowideit

Item5509 IF & query String matching may be incorrectly greedy. Waiting for Release 11 Apr 2008 - 01:55 SvenDowideit

Item5501 IF allows does not work correctly if the topic does not exist. Waiting for Release 11 Apr 2008 - 01:41 SvenDowideit

Item5510 initial version of TalkContrib Closed 09 Apr 2008 - 08:11 SvenDowideit

Item5499 TWiki::UI::Resister::changePassword sends login, not cUID to TWiki::Users::setPassword Waiting for Release 04 Apr 2008 - 02:24 SvenDowideit

Item5496 Conclusion: Turn off UTF-8 test case until bug is fixed (4.2 branch only) Waiting for Release 03 Apr 2008 - 02:06 SvenDowideit

Item5495 add twikiBroadcastMessage class div to default & classic skins Waiting for Release 03 Apr 2008 - 00:53 SvenDowideit

Comments (0)

Feb 15 2008

Blead Perl DTrace probes

Tag: dtrace, perl, solaris — Sven Dowideit @ 5:16 pm

I’ve ported my additional DTrace probes to blead perl, adding the following probes

new-sv and del-sv to track Perl allocations,
main-enter and main-exit to show what part of Perl’s execution phase we’re in
load-module-entry and load-module-return to instrument use, require, do.

I’ve also started to document the probes at http://wikis.sun.com/display/DTrace/perl+Provider
To use the patch,

get bleadperl:
1. rsync -avz –exclude .svn/ –delete rsync://ftp.linux.activestate.com/perl-current/ bleadperl
apply the patch:
cd bleadperl ; patch ../bleadperl.diff
run configure with dtrace enabled:
1. ./Configure -de -Dusedevel -Dinc_version_list=none -Dprefix=/usr/local/bleadperl/ -Dldflags=-Dman3ext=3pm -Duseithreads -Duseshrplib -Uversiononly -Dusedtrace -Doptimize=-g
make
make test
make install
run./perl.d (also found at http://distributedinformation.com/DTrace/ ) and then run whatever Perl code you want to instrument.

Comments (0)

Feb 13 2008

Examining ‘use’ and ‘require’ with Perl Dtrace

Tag: dtrace, perl, twiki — Sven Dowideit @ 6:51 pm

Given the following functionally similar Perl modules, I was curious what the internal Perl differences in execution were.

To recap - (from PerlDoc)

Perl use - Imports some semantics into the current package from the named module - It is exactly equivalent to
```
    BEGIN { require Module; Module->import( LIST ); }
```
Perl require - demands that a library file be included if it hasn’t already been included. There is pseudocode there to suggest that a previously required module (successful or not) would just return the previous result

So given a set of modules that don’t define an import() function, one would expect that there would be little difference.

use Module require Module

main.pl

use Module1; useModule2;
use Module1; # add multiple use's to attempt to simulate having many use statements throughout many modules
use Module1;
use Module1;
use Module1;
use Module1;
use Module1;
BEGIN { print "::BEGIN\n"; }

Module1::printit();

Module1.pm

package Module1;

BEGIN {
my $test = ”;

print “Module1::BEGIN\n"; } sub printit { print "Module1::printit\n";

}
1;

Module2.pm

package Module2; use Module1;

BEGIN {
print “Module2::BEGIN\n”;
}

sub printit {
print “Module2::printit\n”;
}

main.pl

require Module1;
require MyOtherBegin;require Module1;
require Module1;
require Module1;
require Module1;
require Module1;
require Module1;
BEGIN { print "::BEGIN\n"; }

MyBegin::printit();

Module1.pm

package Module1;

BEGIN {
my $test = ”;

print “Module1::BEGIN\n"; }

sub printit {
print “Module1::printit\n";

}
1;

Module2.pm

package Module2; require Module1;

BEGIN {
print “Module2::BEGIN\n”;
}

sub printit {
print “Module2::printit\n”;
}

program output

bash-3.00$ /usr/local/bin/perl main.pl Module1::BEGIN Module2::BEGIN ::BEGIN Module1::printit

program output

bash-3.00$ /usr/local/bin/perl main.pl ::BEGIN Module1::BEGIN Module2::BEGIN Module1::printit

dtrace output

== perl ========================================================== perl*::perl_alloc:main-enter perl*::perl_alloc:main-exit, (0/0) (73 uS) perl*::perl_construct:main-enter perl*::perl_construct:main-exit, (12/0) (543 uS) perl*::perl_parse:main-enter >> perl*::Perl_utilize:load-module-start (Module1) >> perl*::Perl_ck_require:load-module-start (Module1) << perl*::Perl_ck_require:load-module-end (Module1.pm) (254/2) (3 uS) --> BEGIN, main.pl --> BEGIN, Module1.pm <-- BEGIN, Module1.pm (1/0) (102 uS) <-- BEGIN, main.pl (60/38) (763 uS) >> perl*::Perl_utilize:load-module-start (Module2) >> perl*::Perl_ck_require:load-module-start (Module2) << perl*::Perl_ck_require:load-module-end (Module2.pm) (271/16) (2 uS) --> BEGIN, main.pl >> perl*::Perl_utilize:load-module-start (Module1) >> perl*::Perl_ck_require:load-module-start (Module1) << perl*::Perl_ck_require:load-module-end (Module1.pm) (31/8) (2 uS) --> BEGIN, Module2.pm <-- BEGIN, Module2.pm (0/0) (4 uS) --> BEGIN, Module2.pm <-- BEGIN, Module2.pm (1/0) (11 uS) <-- BEGIN, main.pl (67/47) (399 uS) >> perl*::Perl_utilize:load-module-start (Module1) >> perl*::Perl_ck_require:load-module-start (Module1) << perl*::Perl_ck_require:load-module-end (Module1.pm) (285/30) (2 uS) --> BEGIN, main.pl <-- BEGIN, main.pl (0/0) (3 uS) >> perl*::Perl_utilize:load-module-start (Module1) >> perl*::Perl_ck_require:load-module-start (Module1) << perl*::Perl_ck_require:load-module-end (Module1.pm) (299/44) (2 uS) --> BEGIN, main.pl <-- BEGIN, main.pl (0/0) (3 uS) >> perl*::Perl_utilize:load-module-start (Module1) >> perl*::Perl_ck_require:load-module-start (Module1) << perl*::Perl_ck_require:load-module-end (Module1.pm) (313/58) (2 uS) --> BEGIN, main.pl <-- BEGIN, main.pl (0/0) (3 uS) >> perl*::Perl_utilize:load-module-start (Module1) >> perl*::Perl_ck_require:load-module-start (Module1) << perl*::Perl_ck_require:load-module-end (Module1.pm) (327/72) (2 uS) --> BEGIN, main.pl <-- BEGIN, main.pl (0/0) (3 uS)>>perl*::Perl_utilize:load-module-start (Module1)>>perl*::Perl_ck_require:load-module-start (Module1)<<perl*::Perl_ck_require:load-module-end (Module1.pm) (341/86) (2 uS) --> BEGIN, main.pl <-- BEGIN, main.pl (0/0) (3 uS)>>perl*::Perl_utilize:load-module-start (Module1)>>perl*::Perl_ck_require:load-module-start (Module1)<<perl*::Perl_ck_require:load-module-end (Module1.pm) (355/100) (2 uS) --> BEGIN, main.pl <-- BEGIN, main.pl (0/0) (3 uS) --> BEGIN, main.pl <-- BEGIN, main.pl (1/0) (10 uS) perl*::perl_parse:main-exit, (373/132) (3464 uS) perl*::perl_run:main-enter --> printit, Module1.pm <-- printit, Module1.pm (0/0) (8 uS) perl*::perl_run:main-exit, (0/0) (44 uS) perl*::perl_destruct:main-enter perl*::perl_destruct:main-exit, (0/5) (20 uS) total, total (0/0) (4177 uS)Subs returned from: count totaltime mintime avgtime maxtime allocs deallocs func file 1 8 8 8 8 0 0 printit Module1.pm 1 102 102 102 102 1 0 BEGIN Module1.pm 2 15 4 7 11 1 0 BEGIN Module2.pm 9 1190 3 132 763 128 85 BEGIN main.pl count totaltime mintime avgtime maxtime allocs deallocs func file

allocations / deallocations (complete program):
(515 / 222) perl

time to run perl: 6389 uS (time on CPU 4177 uS)

dtrace output

== perl ========================================================== perl*::perl_alloc:main-enter perl*::perl_alloc:main-exit, (0/0) (71 uS) perl*::perl_construct:main-enter perl*::perl_construct:main-exit, (12/0) (583 uS) perl*::perl_parse:main-enter >> perl*::Perl_ck_require:load-module-start (Module1) << perl*::Perl_ck_require:load-module-end (Module1.pm) (248/2) (3 uS)>>perl*::Perl_ck_require:load-module-start (Module2)<<perl*::Perl_ck_require:load-module-end (Module2.pm) (252/2) (2 uS)>>perl*::Perl_ck_require:load-module-start (Module1)<<perl*::Perl_ck_require:load-module-end (Module1.pm) (253/2) (1 uS)>>perl*::Perl_ck_require:load-module-start (Module1)<<perl*::Perl_ck_require:load-module-end (Module1.pm) (254/2) (2 uS)>>perl*::Perl_ck_require:load-module-start (Module1)<<perl*::Perl_ck_require:load-module-end (Module1.pm) (255/2) (1 uS)>>perl*::Perl_ck_require:load-module-start (Module1)<<perl*::Perl_ck_require:load-module-end (Module1.pm) (256/2) (1 uS)>>perl*::Perl_ck_require:load-module-start (Module1)<<perl*::Perl_ck_require:load-module-end (Module1.pm) (257/2) (2 uS)>>perl*::Perl_ck_require:load-module-start (Module1)<<perl*::Perl_ck_require:load-module-end (Module1.pm) (258/2) (1 uS) --> BEGIN, main.pl <-- BEGIN, main.pl (1/0) (100 uS) perl*::perl_parse:main-exit, (285/28) (2319 uS) perl*::perl_run:main-enter --> BEGIN, Module1.pm <-- BEGIN, Module1.pm (1/0) (14 uS) >> perl*::Perl_ck_require:load-module-start (Module1)<<perl*::Perl_ck_require:load-module-end (Module1.pm) (69/46) (2 uS) --> BEGIN, Module2.pm <-- BEGIN, Module2.pm (1/0) (10 uS) --> printit, Module1.pm <-- printit, Module1.pm (0/0) (8 uS) perl*::perl_run:main-exit, (99/73) (772 uS) perl*::perl_destruct:main-enter perl*::perl_destruct:main-exit, (0/13) (36 uS) total, total (0/0) (3814 uS)Subs returned from: count totaltime mintime avgtime maxtime allocs deallocs func file 1 8 0 8 8 0 0 printit Module1.pm 1 10 0 10 10 1 0 BEGIN Module2.pm 1 14 0 14 14 1 0 BEGIN Module1.pm 1 100 0 100 100 1 0 BEGIN main.pl count totaltime mintime avgtime maxtime allocs deallocs func file

allocations / deallocations (complete program):
(399 / 114) perl

time to run perl: 5368 uS (time on CPU 3814 uS)

The (1/0) tuples in braces are counts of allocations and deallocations by the Perl interpreter in in that enter-return pair, and execution times are all using DTrace’s vtimestamp, so are adjusted for time on CPU.

This suggests to me that Perl use and require suffer from the same problems that are encountered when using #include’s inside header files in large scale C and C++, a massive, un-realized parser mess, due to the re-importation of dependencies for the short term convenience of the developer.

While the require output appears better, none the less (looking at the source code for Perl_ck_require) it does more than a nop the second time around (at minimum, its doing a single allocation, and converting the Module name to a File.pm). It is interesting (scary really) that the use case has many more allocations than the require case, but I presume that has to do with the sillyness of importing code into the same namespace several times. That main.pl’s BEGIN is parsed&allocated for 9 times, and Module2:BEGIN is parsed&allocated for twice - is not really what I had in mind.

For those of us working with large scale Perl code (in my case TWiki and Catalyst) it seems that there is not enough clarity, and there are opposing concerns for users that run Perl directly, incurring the parse and allocate steps each time, and those using accelerators such as mod_perl and speedy_cgi, who essentially start at the main-run phase (I think).

Reducing the above examples by removing the use or requires from everywhere except the top level main.pl, still shows an unexpected side effect that the use case has 34 more allocations and 26 deallocations - but… there is less going on. So it is possible, that the inconvenience of needing to manage the Perl module loading at the top level might be worth while, at least until Perl is changed to do what the ‘manual’ implies.

Another interesting side effect of replacing use with require, is that it moves the execution of the BEGIN from the parse phase to the execution phase - again reducing the effectiveness of Perl accelerators, but perhaps more inline with simplistic expectations.

Seems to me its time to move to bleadperl, and see if I can offer a fix.

The above tests are done using the dtrace enabled Perl 5.8.8 that is in my svn repository.

Comments (0)

« Previous Page — Next Page »

DistributedInformation

Debian TWiki repository now with 212 TWiki Plugins, Contribs, Skins and more.

defense against the dark arts? (Cross site scripting and Cross Site forgery)

Firefox 3 pre release builds includes 64bit

progress towards TWiki 4.2.1 patch release.

Blead Perl DTrace probes

Examining ‘use’ and ‘require’ with Perl Dtrace

main.pl

Module1.pm

Module2.pm

main.pl

Module1.pm

Module2.pm

program output

program output

dtrace output

dtrace output