I usually download the tar.gz for a perl module, then it's a simple 'perl Makefile.pl && make && make test'. Install honors DESTDIR, so then I can package it myself, every time. If my distro ever does release an updated module, my package system should pick it up then.
Gems on the other hand, I haven't been able to package at all. Best solution I've seen is Debian, they set up a 'quarantine' under /var/lib/gems, with it's own bin directory and everything, to keep gems away from Debian packaged ruby libs. Then you get to fight with vendoring, config.gem, initializers, etc. I got to the point on one app where I just gave up and copied the libraries into RAILS_ROOT/lib. I sure hope rails 3.0 improves this.