Articles on Perl.com - programming news, code and culture

Making an Asynchronous Clocking Drum Machine App in Perl

Mon, 13 Apr 2026 22:00:00 +0000

Let’s Make a Drum Machine application! Yeah! :D

There are basically two important things to handle: A MIDI “clock” and a groove to play.

Why asynchronous? Well, a simple while (1) { Time::HiRes::sleep($interval); ... } will not do because the time between ticks will fluctuate, often dramatically. IO::Async::Timer::Periodic is a great timer for this purpose. Its default scheduler uses system time, so intervals happen as close to the correct real-world time as possible.

Clocks

A MIDI clock tells a MIDI device about the tempo. This can be handed to a drum machine or a sequencer. Each clock tick tells the device to advance a step of a measured interval. Usually this is very short, and is often 24 pulses per quarter-note (four quarter-notes to a measure of four beats).

Here is code to do that, followed by an explanation of the parts:

#!/usr/bin/env perl

use v5.36;
use feature 'try';
use IO::Async::Loop ();
use IO::Async::Timer::Periodic ();
use MIDI::RtMidi::FFI::Device ();

my $name = shift || 'usb'; # MIDI sequencer device
my $bpm  = shift || 120; # beats per minute

my $interval = 60 / $bpm / 24; # time / bpm / clocks-per-beat

# open the named midi device for output
my $midi_out = RtMidiOut->new;
try { # this will die on Windows but is needed for Mac
    $midi_out->open_virtual_port('RtMidiOut');
}
catch ($e) {}
$midi_out->open_port_by_name(qr/\Q$name/i);

$midi_out->start; # start the sequencer

$SIG{INT} = sub { # halt gracefully
    say "\nStop";
    try {
        $midi_out->stop; # stop the sequencer
        $midi_out->panic; # make sure all notes are off
    }
    catch ($e) {
        warn "Can't halt the MIDI out device: $e\n";
    }
    exit;
};

my $loop = IO::Async::Loop->new;

my $timer = IO::Async::Timer::Periodic->new(
   interval => $interval,
   on_tick  => sub { $midi_out->clock }, # send a clock tick!
);
$timer->start;

$loop->add($timer);
$loop->run;

The above code does a few things. First it uses modern Perl, then the modules that will make execution asynchronous, and finally the module that makes real-time MIDI possible.

Next up, a $name variable is captured for a unique MIDI device. (And to see what the names of MIDI devices on the system are, use JBARRETT’s little list_devices script.) Also, the beats per minute is taken from the command-line. If neither is given, usb is used for the name, and the BPM is set to “dance tempo.”

The clock needs a time interval to tick off. For us, this is a fraction of a second based on the beats per minute, and is assigned to the $interval variable.

To get the job done, we will need to open the named MIDI device for sending output messages to. This is done with the $name provided.

In order to not just die when we want to stop, $SIG{INT} is redefined to gracefully halt. This also sends a stop message to the open MIDI device. This stops the sequencer from playing.

Now for the meat and potatoes: The asynchronous loop and periodic timer. These tell the program to do its thing, in a non-blocking and event-driven manner. The periodic timer ticks off a clock message every $interval. Pretty simple!

As an example, here is the above code controlling my Volca Drum drum machine on a stock, funky groove. We invoke it on the command-line like this:

perl clock-gen-async.pl

Your browser does not support the audio element.

Grooves

What we really want is to make our drum machine actually play something of our own making. So it’s refactor time… Let’s make a 4/4 time groove, with 16th-note resolution, that alternates between two different parts. “4/4” is a “time signature” in music jargon and means that there are four beats per measure (numerator), and a quarter note equals one beat (denominator). Other time signatures like the waltz’s 3/4 are simple, while odd meters like 7/8 are not.

In order to generate syncopated patterns, Math::Prime::XS and Music::CreatingRhythms are added to the use statements. “What are syncopated patterns?”, you may ask. Good question! “Syncopated” means, “characterized by displaced beats.” That is, every beat does not happen evenly, at exactly the same time. Instead, some are displaced. For example, a repeated [1 1 1 1] is even and boring. But when it becomes a repeated [1 1 0 1] things get spicier and more syncopated.

The desired MIDI channel is added to the command-line inputs. Most commonly, this will be channel 9 (in zero-based numbering). But some drum machines and sequencers are “multi-timbral” and use multiple channels simultaneously for individual sounds.

Next we define the drums to use. This is a hash-reference that includes the MIDI patch number, the channel it’s on, and the pattern to play. The combined patterns of all the drums, when played together at tempo, make a groove.

Now we compute intervals and friends. Previously, there was one $interval. Now there are a whole host of measurements to make before sending MIDI messages.

Then, as before, a named MIDI output device is opened, and a graceful stop is defined.

Next, a Music::CreatingRhythms object is created. And then, again as before, an asynchronous loop and periodic timer are instantiated and set in motion.

The meaty bits are in the timer’s on_tick callback. This contains all the logic needed to trigger our drum grooves.

As was done in the previous clock code, a clock message is sent, but also we keep track of the number of clock ticks that have passed. This number of ticks is used to trigger the drums. We care about 16 beats. So every 16th beat, we construct and play a queue of events.

Adjusting the drum patterns is where Math::Prime::XS and Music::CreatingRhythms come into play. The subroutine that does that is adjust_drums() and is fired every 4th measure. A measure is equal to four quarter-notes, and we use four pulses for each, to make 16 beats per measure. This routine reassigns either Euclidean or manual patterns of 16 beats to each drum pattern.

Managing the queue is next. If a drum is to be played at the current beat (as tallied by the $beat_count variable), it is added to the queue at full velocity (127). Then, after all the drums have been accounted for, the queue is played with $midi_out->note_on() messages. Lastly, the queue is “drained” by sending $midi_out->note_off() messages.

#!/usr/bin/env perl

use v5.36;
use feature 'try';
use IO::Async::Loop ();
use IO::Async::Timer::Periodic ();
use Math::Prime::XS qw(primes);
use MIDI::RtMidi::FFI::Device ();
use Music::CreatingRhythms ();

my $name = shift || 'usb'; # MIDI sequencer device
my $bpm  = shift || 120; # beats-per-minute
my $chan = shift // 9; # 0-15, 9=percussion, -1=multi-timbral

my $drums = {
    kick  => { num => 36, chan => $chan < 0 ? 0 : $chan, pat => [] },
    snare => { num => 38, chan => $chan < 0 ? 1 : $chan, pat => [] },
    hihat => { num => 42, chan => $chan < 0 ? 2 : $chan, pat => [] },
};

my $beats = 16; # beats in a measure
my $divisions = 4; # divisions of a quarter-note into 16ths
my $clocks_per_beat = 24; # PPQN
my $clock_interval = 60 / $bpm / $clocks_per_beat; # time / bpm / ppqn
my $sixteenth = $clocks_per_beat / $divisions; # clocks per 16th-note
my %primes = ( # for computing the pattern
    all  => [ primes($beats) ],
    to_5 => [ primes(5) ],
    to_7 => [ primes(7) ],
);
my $ticks = 0; # clock ticks
my $beat_count = 0; # how many beats?
my $toggle = 0; # part A or B?
my @queue; # priority queue for note_on/off messages

# open the named midi output device
my $midi_out = RtMidiOut->new;
try { # this will die on Windows but is needed for Mac
    $midi_out->open_virtual_port('RtMidiOut');
}
catch ($e) {}
$midi_out->open_port_by_name(qr/\Q$name/i);

$SIG{INT} = sub { # halt gracefully
    say "\nStop";
    try {
        $midi_out->stop; # stop the sequencer
        $midi_out->panic; # make sure all notes are off
    }
    catch ($e) {
        warn "Can't halt the MIDI out device: $e\n";
    }
    exit;
};

# for computing the pattern
my $mcr = Music::CreatingRhythms->new;

my $loop = IO::Async::Loop->new;

my $timer = IO::Async::Timer::Periodic->new(
    interval => $clock_interval,
    on_tick  => sub {
        $midi_out->clock;
        $ticks++;
        if ($ticks % $sixteenth == 0) {
            # adjust the drum pattern every 4th measure
            if ($beat_count % ($beats * $divisions) == 0) {
                adjust_drums($mcr, $drums, \%primes, \$toggle);
            }
            # add simultaneous drums to the queue
            for my $drum (keys %$drums) {
                if ($drums->{$drum}{pat}[ $beat_count % $beats ]) {
                    push @queue, { drum => $drum, velocity => 127 };
                }
            }
            # play the queue
            for my $drum (@queue) {
                $midi_out->note_on(
                    $drums->{ $drum->{drum} }{chan},
                    $drums->{ $drum->{drum} }{num},
                    $drum->{velocity}
                );
            }
            $beat_count++;
        }
        else {
            # drain the queue with note_off messages
            while (my $drum = pop @queue) {
                $midi_out->note_off(
                    $drums->{ $drum->{drum} }{chan},
                    $drums->{ $drum->{drum} }{num},
                    0
                );
            }
            @queue = (); # ensure the queue is empty
        }
    },
);
$timer->start;

$loop->add($timer);
$loop->run;

sub adjust_drums($mcr, $drums, $primes, $toggle) {
    # choose random primes to use by the hihat, kick, and snare
    my ($p, $q, $r) = map { $primes->{$_}[ int rand $primes->{$_}->@* ] } sort keys %$primes;
    if ($$toggle == 0) {
        say 'part A';
        $drums->{hihat}{pat} = $mcr->euclid($p, $beats);
        $drums->{kick}{pat}  = $mcr->euclid($q, $beats);
        $drums->{snare}{pat} = $mcr->rotate_n($r, $mcr->euclid(2, $beats));
        $$toggle = 1; # set to part B
    }
    else {
        say 'part B';
        $drums->{hihat}{pat} = $mcr->euclid($p, $beats);
        $drums->{kick}{pat}  = [qw(1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1)];
        $drums->{snare}{pat} = [qw(0 0 0 0 1 0 0 0 0 0 0 0 1 0 1 0)];
        $$toggle = 0; # set to part A
    }
}

(You may notice the inefficiency of attempting to drain an empty queue 23 times every 16th note. Oof! Fortunately, this doesn’t fire anything other than a single while loop condition. A more efficient solution would be to only drain the queue once, but this requires a bit more complexity that we won’t be adding, for brevity’s sake.)

On Windows, this works fine:

perl clocked-euclidean-drums.pl "gs wavetable" 90

To run with fluidsynth and hear the General MIDI percussion sounds, open a fresh new terminal session, and start up fluidsynth like so (mac syntax):

fluidsynth -a coreaudio -m coremidi -g 2.0 ~/Music/soundfont/FluidR3_GM.sf2

The FluidR3_GM.sf2 is a MIDI “soundfont” file and can be downloaded for free.

Next, enter this on the command-line (back in the previous terminal session):

perl clocked-euclidean-drums.pl fluid 90

You will hear standard kick, snare, and closed hihat cymbal. And here is a poor recording of this with my phone:

Your browser does not support the audio element.

To run the code with my multi-timbral drum machine, I enter this on the command-line:

perl clocked-euclidean-drums.pl usb 90 -1

And here is what that sounds like:

Your browser does not support the audio element.

The Module

I have coded this logic, and a bit more, into a friendly CPAN module. Check out the eg/euclidean.pl example program in the distribution. It is a work in progress. YMMV.

Credits

Thank you to Andrew Rodland (hobbs), who helped me wrap my head around the “no-sleeping asynchronous” algorithm.

To-do Challenges

Make patterns other than prime number based Euclidean phrases.
Toggle more than two groove parts.
Add snare fills to the (end of the) 4th bars. (here’s my version)
Make this code handle odd meter grooves.

Resources

The IO::Async::Loop module
The IO::Async::Timer::Periodic module
The Math::Prime::XS module
The MIDI::RtMidi::FFI::Device module
The Music::CreatingRhythms module
The Music::SimpleDrumMachine WIP module based on this logic
The cross-platform fluidsynth application
My original music: https://www.youtube.com/@GeneBoggs

Musical Rhythms with Math in Perl

Sun, 15 Mar 2026 00:00:00 +0000

Let’s talk about music programming! There are a million aspects to this subject, but today, we’ll touch on generating rhythmic patterns with mathematical and combinatorial techniques. These include the generation of partitions, necklaces, and Euclidean patterns.

Stefan and J. Richard Hollos wrote an excellent little book called “Creating Rhythms” that has been turned into C, Perl, and Python. It features a number of algorithms that produce or modify lists of numbers or bit-vectors (of ones and zeroes). These can be beat onsets (the ones) and rests (the zeroes) of a rhythm. We’ll check out these concepts with Perl.

For each example, we’ll save the MIDI with the MIDI::Util module. Also, in order to actually hear the rhythms, we will need a MIDI synthesizer. For these illustrations, fluidsynth will work. Of course, any MIDI capable synth will do! I often control my eurorack analog synthesizer with code (and a MIDI interface module).

Here’s how I start fluidsynth on my mac in the terminal, in a separate session. It uses a generic soundfont file (sf2) that can be downloaded here (124MB zip).

fluidsynth -a coreaudio -m coremidi -g 2.0 ~/Music/soundfont/FluidR3_GM.sf2

So, how does Perl know what output port to use? There are a few ways, but with JBARRETT’s MIDI::RtMidi::FFI::Device, you can do this:

use MIDI::RtMidi::FFI::Device ();

my $midi_in = RtMidiIn->new;
my $midi_out = RtMidiOut->new;

print "Input devices:\n";
$midi_in->print_ports;
print "\n";

print "Output devices:\n";
$midi_out->print_ports;
print "\n";

This shows that fluidsynth is alive and ready for interaction.

Okay, on with the show!

First-up, let’s look at partition algorithms. With the part() function, we can generate all partitions of n, where n is 5, and the “parts” all add up to 5. Then taking one of these (say, the third element), we convert it to a binary sequence that can be interpreted as a rhythmic phrase, and play it 4 times.

#!/usr/bin/env perl
use strict;
use warnings;

use Music::CreatingRhythms ();

my $mcr = Music::CreatingRhythms->new;

my $parts = $mcr->part(5);
# [ [ 1, 1, 1, 1, 1 ], [ 1, 1, 1, 2 ], [ 1, 2, 2 ], [ 1, 1, 3 ], [ 2, 3 ], [ 1, 4 ], [ 5 ] ]

my $p = $parts->[2]; # [ 1, 2, 2 ]

my $seq = $mcr->int2b([$p]); # [ [ 1, 1, 0, 1, 0 ] ]

Now we render and save the rhythm:

use MIDI::Util qw(setup_score);

my $score = setup_score(bpm => 120, channel => 9);

for (1 .. 4) {
    for my $bit ($seq->[0]->@*) {
        if ($bit) {
            $score->n('en', 40);
        }
        else {
            $score->r('en');
        }
    }
}

$score->write_score('perldotcom-1.mid');

In order to play the MIDI file that is produced, we can use fluidsynth like this:

fluidsynth -i ~/Music/soundfont/FluidR3_GM.sf2 perldotcom-1.mid

Your browser does not support the audio element.

Not terribly exciting yet.

Let’s see what the “compositions” of a number reveal. According to the Music::CreatingRhythms docs, a composition of a number is “the set of combinatorial variations of the partitions of n with the duplicates removed.”

Okay. Well, the 7 partitions of 5 are:

[[1, 1, 1, 1, 1], [1, 1, 1, 2], [1, 1, 3], [1, 2, 2], [1, 4], [2, 3], [5]]

And the 16 compositions of 5 are:

[[1, 1, 1, 1, 1], [1, 1, 1, 2], [1, 1, 2, 1], [1, 1, 3], [1, 2, 1, 1], [1, 2, 2], [1, 3, 1], [1, 4], [2, 1, 1, 1], [2, 1, 2], [2, 2, 1], [2, 3], [3, 1, 1], [3, 2], [4, 1], [5]]

That is, the list of compositions has, not only the partition [1, 2, 2], but also its variations: [2, 1, 2] and [2, 2, 1]. Same with the other partitions. Selections from this list will produce possibly cool rhythms.

Here are the compositions of 5 turned into sequences, played by a snare drum, and written to the disk:

use Music::CreatingRhythms ();
use MIDI::Util qw(setup_score);

my $mcr = Music::CreatingRhythms->new;

my $comps = $mcr->compm(5, 3); # compositions of 5 with 3 elements

my $seq = $mcr->int2b($comps);

my $score = setup_score(bpm => 120, channel => 9);

for my $pattern ($seq->@*) {
    for my $bit (@$pattern) {
        if ($bit) {
            $score->n('en', 40); # snare patch
        }
        else {
            $score->r('en');
        }
    }
}

$score->write_score('perldotcom-2.mid');

Your browser does not support the audio element.

A little better. Like a syncopated snare solo.

Another way to play the MIDI file is to use timidity. On my mac, with the soundfont specified in the timidity.cfg configuration file, this would be:

timidity -c ~/timidity.cfg -Od perldotcom-2.mid

To convert a MIDI file to an mp3 (or other audio formats), I do this:

timidity -c ~/timidity.cfg perldotcom-2.mid -Ow -o - | ffmpeg -i - -acodec libmp3lame -ab 64k perldotcom-2.mp3

Okay. Enough technical details! What if we want a kick bass drum and hi-hat cymbals, too? Refactor time…

use MIDI::Util qw(setup_score);
use Music::CreatingRhythms ();

my $mcr = Music::CreatingRhythms->new;

my $s_comps = $mcr->compm(4, 2); # snare
my $s_seq = $mcr->int2b($s_comps);

my $k_comps = $mcr->compm(4, 3); # kick
my $k_seq = $mcr->int2b($k_comps);

my $score = setup_score(bpm => 120, channel => 9);

for (1 .. 8) { # repeats
    my $s_choice = $s_seq->[ int rand @$s_seq ];
    my $k_choice = $k_seq->[ int rand @$k_seq ];

    for my $i (0 .. $#$s_choice) { # pattern position
        my @notes = (42); # hi-hat every time
        if ($s_choice->[$i]) {
            push @notes, 40;
        }
        if ($k_choice->[$i]) {
            push @notes, 36;
        }
        $score->n('en', @notes);
    }
}

$score->write_score('perldotcom-3.mid');

Your browser does not support the audio element.

Here we play generated kick and snare patterns, along with a steady hi-hat.

Next up, let’s look at rhythmic “necklaces.” Here we find many grooves of the world.

Image from The Geometry of Musical Rhythm

Rhythm necklaces are circular diagrams of equally spaced, connected nodes. A necklace is a lexicographical ordering with no rotational duplicates. For instance, the necklaces of 3 beats are [[1, 1, 1], [1, 1, 0], [1, 0, 0], [0, 0, 0]]. Notice that there is no [1, 0, 1] or [0, 1, 1]. Also, there are no rotated versions of [1, 0, 0], either.

So, how many 16 beat rhythm necklaces are there?

my $necklaces = $mcr->neck(16);
print scalar @$necklaces, "\n"; # 4116 of 'em!

Okay. Let’s generate necklaces of 8 instead, pull a random choice, and play the pattern with a percussion instrument.

use MIDI::Util qw(setup_score);
use Music::CreatingRhythms ();

my $patch = shift || 75; # claves

my $mcr = Music::CreatingRhythms->new;

my $necklaces = $mcr->neck(8);
my $choice = $necklaces->[ int rand @$necklaces ];

my $score = setup_score(bpm => 120, channel => 9);

for (1 .. 4) { # repeats
    for my $bit (@$choice) { # pattern position
        if ($bit) {
            $score->n('en', $patch);
        }
        else {
            $score->r('en');
        }
    }
}

$score->write_score('perldotcom-4.mid');

Your browser does not support the audio element.

Here we choose from all necklaces. But note that this also includes the sequence with all ones and the sequence with all zeroes. More sophisticated code might skip these.

More interesting would be playing simultaneous beats.

use MIDI::Util qw(setup_score);
use Music::CreatingRhythms ();

my $mcr = Music::CreatingRhythms->new;

my $necklaces = $mcr->neck(8);

my $x_choice = $necklaces->[ int rand @$necklaces ];
my $y_choice = $necklaces->[ int rand @$necklaces ];
my $z_choice = $necklaces->[ int rand @$necklaces ];

my $score = setup_score(bpm => 120, channel => 9);

for (1 .. 4) { # repeats
    for my $i (0 .. $#$x_choice) { # pattern position
        my @notes;
        if ($x_choice->[$i]) {
            push @notes, 75; # claves
        }
        if ($y_choice->[$i]) {
            push @notes, 63; # hi_conga
        }
        if ($z_choice->[$i]) {
            push @notes, 64; # low_conga
        }
        $score->n('en', @notes);
    }
}

$score->write_score('perldotcom-5.mid');

And that sounds like:

Your browser does not support the audio element.

How about Euclidean patterns? What are they, and why are they named for a geometer?

Euclidean patterns are a set number of positions P that are filled with a number of beats Q that is less than or equal to P. They are named for Euclid because they are generated by applying the “Euclidean algorithm,” which was originally designed to find the greatest common divisor (GCD) of two numbers, to distribute musical beats as evenly as possible.

use MIDI::Util qw(setup_score);
use Music::CreatingRhythms ();

my $mcr = Music::CreatingRhythms->new;

my $beats = 16;

my $s_seq = $mcr->rotate_n(4, $mcr->euclid(2, $beats)); # snare
my $k_seq = $mcr->euclid(2, $beats); # kick
my $h_seq = $mcr->euclid(11, $beats); # hi-hats

my $score = setup_score(bpm => 120, channel => 9);

for (1 .. 4) { # repeats
    for my $i (0 .. $beats - 1) { # pattern position
        my @notes;
        if ($s_seq->[$i]) {
            push @notes, 40; # snare
        }
        if ($k_seq->[$i]) {
            push @notes, 36; # kick
        }
        if ($h_seq->[$i]) {
            push @notes, 42; # hi-hats
        }
        if (@notes) {
            $score->n('en', @notes);
        }
        else {
            $score->r('en');
        }
    }
}

$score->write_score('perldotcom-6.mid');

Your browser does not support the audio element.

Now we’re talkin’ - an actual drum groove! To reiterate, the euclid() method distributes a number of beats, like 2 or 11, over the number of beats, 16. The kick and snare use the same arguments, but the snare pattern is rotated by 4 beats, so that they alternate.

So what have we learned today?

That you can use mathematical functions to generate sequences to represent rhythmic patterns.
That you can play an entire sequence or simultaneous notes with MIDI.

References:

Fastmail Donates USD 10,000 to The Perl and Raku Foundation

Wed, 04 Feb 2026 00:45:55 +0000

2025 was a tough year for The Perl and Raku Foundation (TPRF). Funds were sorely needed. The community grants program had been paused due to budget constraints and we were in danger of needing to pause the Perl 5 core maintenance grants. Fastmail stepped up with a USD 10,000 donation and helped TPRF to continue to support Perl 5 core maintenance. Ricardo Signes explains why Fastmail helped keep this very important work on track.

Perl has served us quite well since Fastmail’s inception. We’ve built up a large code base that has continued to work, grow, and improve over twenty years. We’ve stuck with Perl because Perl stuck with us: it kept working and growing and improving, and very rarely did those improvements require us to stop the world and adapt to onerous changes. We know that kind of stability is, in part, a function of the developers of Perl, whose time is spent figuring out how to make Perl better without also making it worse. The money we give toward those efforts is well-spent, because it keeps the improvements coming and the language reliable.

— Ricardo Signes, Director & Chief Developer Experience Officer, Fastmail

One of the reasons that you don’t hear about Perl in the headlines is its reliability. Upgrading your Perl from one version to the next? That can be a very boring deployment. You code worked before and it continues to “just work” after the upgrade. You don’t need to rant about short deprecation cycles, performance degradation or dependencies which no longer install. The Perl 5 core maintainers take great care to ensure that you don’t have to care very much about upgrading your Perl. Backwards compatibility is top of mind. If your deployment is boring, it’s because a lot of care and attention has been given to this matter by the people who love Perl and love to work on it.

As we moved to secure TPRF’s 2025 budget, we reached out to organizations which rely on Perl. A number of these companies immediately offered to help. Fastmail has already been a supporter of TPRF for quite some time. In addition to this much needed donation, Fastmail has been providing rock solid free email hosting to the foundation for many years.

While Fastmail’s donation has been allocated towards Perl 5 Core maintenance, TPRF is now in the position to re-open the community grants program, funding it with USD 10,000 for 2026. There is also an opportunity to increase the community grants funding if sponsor participation increases. As we begin our 2026 fundraising, we are looking to cast a wider net and bring more sponsor organizations on board to help support healthy Perl and Raku ecosystems.

Maybe your organization will be the one to help us double our community grants budget in 2026. To become a sponsor, contact: olaf@perlfoundation.org

Cast-Iron Community: Your Chance to Sponsor TPRC 2026

Tue, 03 Feb 2026 00:00:00 +0000

“Perl is my cast-iron pan - reliable, versatile, durable, and continues to be ever so useful.” TPRC 2026 brings together a community that embodies all of these qualities, and we’re looking for sponsors to help make this special gathering possible.

About the Conference

The Perl and Raku Conference 2026 is a community-organized gathering of developers, enthusiasts, and industry professionals. It takes place from June 26-28, 2026, in Greenville, South Carolina. The conference will feature an intimate, single-track format that promises high sponsor visibility. We look forward to approximately 80 participants with some of those staying in town for the shoulder days (June 25-29) and a Monday workshop.

Give back to the language and communities which have already given so much to you
Connect with the developers and craftspeople who build your tools – the ones that are built to last
Help to ensure that The Perl and Raku Foundation can continue to fund Perl 5 core maintenance and Community Grants

Sponsorship Tiers

Only 1 sponsorship is available at this level
Premium logo placement on conference website
This donation qualifies your organization to be a Bronze Level Sponsor of The Perl and Raku Foundation
5-minute speaking slot during opening ceremony
2 complimentary conference passes
Priority choice of rollup banner placement
Logo prominently displayed on conference badges
First choice of major named sponsorship (Conference Dinner, T-shirts, or Swag Bags)
Logo on main stage backdrop and conference banners
Social media promotion
All benefits of lower tiers

Logo on all conference materials
One complimentary conference pass
Rollup banner on display
Choice of named sponsorship (Lunch or Snacks)
Logo on backdrop and banners
Dedicated social media recognition
All benefits of lower tiers

Logo on conference website
Logo on backdrop and banners
Choice of smaller named sponsorship (Beverage Bars)
Social media mention
All benefits of lower tier

Name/logo on conference website
Name/logo on backdrop and banners

All Sponsors Receive

Logo/name in Update::Daily conference newsletter sidebar
Opportunity to provide materials for conference swag bags
Recognition during opening and closing ceremonies
Listed on conference website sponsor page
Mentioned in conference social media

Named Sponsorship Opportunities

Exclusive naming rights available for:

Conference Dinner ($2,000) - Signage on tables and buffet
Conference Swag Bags ($1,500) - Logo on bags
Conference T-Shirts ($1,500) - Logo on sleeve
Lunches ($1,500) - Signage at pickup and on menu tickets
Snacks ($1,000) - Signage at snack bar
Update::Daily Printing ($200) - Logo on masthead

About The Perl and Raku Foundation

Proceeds beyond conference expenses support The Perl and Raku Foundation, a non-profit organization dedicated to advancing the Perl and Raku programming languages through open source development, education, and community building.

Contact Information

For more information on how to become a sponsor, please contact: olaf@perlfoundation.org

Podlite comes to Perl: a lightweight block-based markup language for everyday use

Tue, 27 Jan 2026 18:13:16 +0000

My name is Alex. Over the last years I’ve implemented several versions of the Raku’s documentation format (Synopsys 26 / Raku’s Pod) in Perl and JavaScript.

At an early stage, I shared the idea of creating a lightweight version of Raku’s Pod, with Damian Conway, the original author of the Synopsys 26 Documentation specification (S26). He was supportive of the concept and offered several valuable insights that helped shape the vision of what later became Podlite.

Today, Podlite is a small block-based markup language that is easy to read as plain text, simple to parse, and flexible enough to be used everywhere — in code, notes, technical documents, long-form writing, and even full documentation systems.

This article is an introduction for the Perl community — what Podlite is, how it looks, how you can already use it in Perl via a source filter, and what’s coming next.

The Block Structure of Podlite

One of the core ideas behind Podlite is its consistent block-based structure. Every meaningful element of a document — a heading, a paragraph, a list item, a table, a code block, a callout — is represented as a block. This makes documents both readable for humans and predictable for tools.

Podlite supports three interchangeable block styles: delimited, paragraph, and abbreviated.

Abbreviated blocks (`=BLOCK`)

This is the most compact form. A block starts with = followed by the block name.

=head1 Installation Guide
=item Perl 5.8 or newer
=para This tool automates the process.

ends on the next directive or a blank line
best used for simple one-line blocks
cannot include configuration options (attributes)

Paragraph blocks (`=for BLOCK`)

Use this form when you want a multi-line block or need attributes.

=for code :lang<perl>
say "Hello from Podlite!";

ends when a blank line appears
can include complex content
allows attributes such as :lang, :id, :caption, :nested, …

Delimited blocks (`=begin BLOCK` … `=end BLOCK`)

The most expressive form. Useful for large sections, nested blocks, or structures that require clarity.

=begin nested :notify<important>
Make sure you have administrator privileges.
=end nested

explicit start and end markers
perfect for code, lists, tables, notifications, markdown, formulas
can contain other blocks, including nested ones

These block styles differ in syntax convenience, but all produce the same internal structure.

Regardless of which syntax you choose:

all three forms represent the same block type
attributes apply the same way (:lang, :caption, :id, …)
tools and renderers treat them uniformly
nested blocks work identically
you can freely mix styles inside a document

Example: Comparing POD and Podlite

Let’s see how the same document looks in traditional POD versus Podlite:

Each block has clear boundaries, so you don’t need blank lines between them. This makes your documentation more compact and easier to read. This is one of the reasons Podlite remains compact yet powerful: the syntax stays flexible, while the underlying document model stays clean and consistent.

This Podlite example rendered as on the following screen:

Inside the Podlite Specification 1.0

One important point about Podlite is that it is first and foremost a specification. It does not belong to any particular programming language, platform, or tooling ecosystem. The specification defines the document model, syntax rules, and semantics.

From the Podlite 1.0 specification, notable features include:

headings (=head1, =head2, …)
lists and definition lists, and including task lists
tables (simple and advanced)
CSV-backed tables
callouts / notifications (=nested :notify<tip|warning|important|note|caution>)
table of contents (=toc)
includes (=include)
embedded data (=data)
pictures (=picture and inline P<>)
formulas (=formula and inline F<>)
user defined blocks and markup codes
Markdown integration

The =markdown block is part of the standard block set defined by the Podlite Specification 1.0. This means Markdown is not an add-on or optional plugin — it is a fully integrated, first-class component of the language.

Markdown content becomes part of Podlite’s unified document structure, and its headings merge naturally with Podlite headings inside the TOC and document outline.

Below is a screenshot showing how Markdown inside Perl is rendered in the in-development VS Code extension, demonstrating both the block structure and live preview:

Using Podlite in Perl via the source filter

To make Podlite directly usable in Perl code, there is a module on CPAN: Podlite — Use Podlite markup language in Perl programs

A minimal example could look like this:

use Podlite; # enable Podlite blocks inside Perl

=head1 Quick Example
=begin markdown
Podlite can live inside your Perl programs.
=end markdown
print "Podlite active\n";

Roadmap: what’s next for Podlite

Podlite continues to grow, and the Specification 1.0 is only the beginning. Several areas are already in active development, and more will evolve with community feedback.

Some of the things currently planned or in progress:

CLI tools
- command-line utilities for converting Podlite to HTML, PDF, man pages, etc.
- improve pipelines for building documentation sites from Podlite sources
VS Code integration
Ecosystem growth
- develop comprehensive documentation and tutorials
- community-driven block types and conventions

If this resonates with you, I’d be very happy to hear from you:

ideas for useful block types
suggestions for tools or integrations
feedback on the syntax and specification

https://github.com/podlite/podlite-specs/discussions

Even small contributions — a comment, a GitHub star, or trying an early tool — help shape the future of the specification and encourage further development.

Useful links:

CPAN: https://metacpan.org/pod/Podlite
GitHub:https://github.com/podlite
Specification
- (HTML): https://podlite.org/specification
- source: https://github.com/podlite/podlite-specs
Project site: https://podlite.org
Roadmap: https://podlite.org/#Roadmap

Thanks for reading, Alex

Announcing the Perl Toolchain Summit 2026

Fri, 23 Jan 2026 12:00:00 +0000

Announcing the Perl Toolchain Summit 2026!

The organizers have been working behind the scenes since last September, and today I’m happy to announce that the 16th Perl Toolchain Summit will be held in Vienna, Austria, from Thursday April 23rd till Sunday April 26th, 2026.

This post is brought to you by Simplelists, a group email and mailing list service provider, and a recurring sponsor of the Perl Toolchain Summit.

Started in 2008 as the Perl QA Hackathon in Oslo, the Perl Toolchain Summit is an annual event that brings together the key developers working on the Perl toolchain. Each year (except for 2020-2022), the event moves from country to country all over Europe, organised by local teams of volunteers. The surplus money from previous summits helps fund the next one.

Since 2023, the organizing team is formally split between a “global” team and a “local” team (although this setup has been informally used before).

The global team is made up of veteran PTS organizers, who deal with invitations, finding sponsors, paying bills and communications. They are Laurent Boivin (ELBEHO), Philippe Bruhat (BOOK), Thibault Duponchelle (CONTRA), Tina Müller (TINITA) and Breno de Oliveira (GARU), supported by Les Mongueurs de Perl’s bank account.

The local team members for this year have organized several events in Vienna (including the Perl QA Hackathon 2010!) and deal with finding the venue, the hotel, the catering and welcoming our attendees in Vienna in April. They are Alexander Hartmaier (ABRAXXA), Thomas Klausner (DOMM), Maroš Kollár (MAROS), Michael Kröll and Helmut Wollmersdorfer (WOLLMERS).

The developers who maintain CPAN and associated tools and services are all volunteers, scattered across the globe. This event is the one time in the year when they can get together.

The summit provides dedicated time to work on the critical systems and tools, with all the right people in the same room. The attendees hammer out solutions to thorny problems and discuss new ideas to keep the toolchain moving forward. This year, about 40 people have been invited, with 35 participants expected to join us in Vienna.

If you want to find out more about the work being done at the Toolchain Summit, and hear the teams and people involved, you can listen to several episodes of The Underbar podcast, which were recorded during the 2025 edition in Leipzig, Germany:

Given the important nature of the attendees’ work and their volunteer status, we try to pay for most expenses (travel, lodging, food, etc.) through sponsorship. If you’re interested in helping sponsor the summit, please get in touch with the global team at pts2026@perltoolchainsummit.org.

Simplelists has been sponsoring the Perl Toolchain Summit for several years now. We are very grateful for their continued support.

Simplelists is proud to sponsor the 2026 Perl Toolchain Summit, as Perl forms the core of our technology stack. We are grateful that we can rely on the robust and comprehensive Perl ecosystem, from the core of Perl itself to a whole myriad of CPAN modules. We are glad that the PTS continues its unsung work, ensuring that Simplelists can continue to rely on these many tools.

DuckDuckGo Donates $25,000 to The Perl and Raku Foundation v2025

Wed, 01 Oct 2025 12:00:00 +0000

For the second consecutive year, The Perl and Raku Foundation (TPRF) is overjoyed to announce a donation of USD 25,000 from DuckDuckGo.

DuckDuckGo has demonstrated how Perl and its ecosystem can deliver power and scale to drive the DuckDuckGo core systems, plug-in framework and Instant Answers. The Foundation is grateful that DuckDuckGo recognises the importance of Perl, and for their generous funding support for a second year through their charitable donations programme.

– Stuart J Mackintosh, President of The Perl and Raku Foundation

Last year’s donation of USD 25,000 from DuckDuckGo was instrumental in helping to fund the foundation’s Core Perl Maintenance Fund and this year’s donation will help to fund more of the same crucial work that keeps the Perl language moving forward.

Paul “LeoNerd” Evans is one of the developers who gets regular funding from the Perl Core Maintenance Fund. Here is a short list of just some of the many contributions which Paul has made to core Perl as part of the maintenance fund work:

The builtin module (5.36), making available many new useful language-level utilities that were previously loaded from modules like Scalar::Util
The complete feature ‘class’ system (5.38), adding proper object-orientation syntax and abilities
Lexical method support (5.42), adding my method and the $obj->&method invocation syntax for better object encapsulation
Stabilising some of the recent experiments - signatures (5.36), try/catch (5.40), foreach on multiple vars (5.40)
Ability to use the //= and ||= operators in signatures (5.38), performance improvements and named parameters (upcoming in next release)
The new any and all keywords (5.42)

We look forward to many more innovative contributions from Paul over the coming year.

While TPRF never takes continued support for granted, when it does arrive, it allows the foundation to plan for the future with much greater confidence. Multi-year partnerships with our sponsors allow us to continue to prioritize important work, knowing that we will have the runway that we need to fund the work which helps to sustain the Perl Language and its associated communities.

For more information on how to become a sponsor, please contact: olaf@perlfoundation.org

"Fireworks" by colink. is licensed under CC BY-SA 2.0.

Geizhals Preisvergleich Donates USD 10,000 to The Perl and Raku Foundation

Thu, 18 Sep 2025 09:30:00 +0000

Today The Perl and Raku Foundation is thrilled to announce a donation of USD 10,000 from Geizhals Preisvergleich. This gift helps to secure the future of The Perl 5 Core Maintenance Fund.

Perl has been an integral part of our product price comparison platform from the start of the company 25 years ago. Supporting the Perl 5 Core Maintenance Fund means supporting both present and future of a substantial pillar of Modern Open Source Computing, for us and other current or prospective users.

– Michael Kröll of Geizhals Preisvergleich

“Geizhals is not only providing core funding for the Perl ecosystem, but also supporting developers, actively contributing to European conferences, and employing Perl coders. Their interest in the strategic maintenance and development of Perl and CPAN is of great value to us all, and their investment is very much appreciated.”

– Stuart J Mackintosh, President of The Perl and Raku Foundation

But who exactly is Geizhals, and why does their support matter so much to the Perl community?

Geizhals Preisvergleich began in July of 1997 as a hobby project—and yes, “Geizhals” literally translates to “skinflint” in English (they even operate skinflint.co.uk for UK users!). From those humble beginnings, they’ve leveraged the power of Perl to scale up to serving 4.3 million monthly users. With Perl being a key part of their infrastructure, they have generously decided to support the Perl 5 Core Maintenance Fund.

While many of us know about the Core Maintenance Fund, the specific problems it addresses often remain invisible to users. I reached out to the maintainers whose work is supported by this fund. This is what core maintainer Tony Cook had to say:

My work tends to be little things, I review other people’s work which I think improves quality and velocity, and fix more minor issues, some examples would be:

a fix to signal handling where perl could crash where an external library created threads (#22487)

fix a segmentation fault in smartmatch against a sub if the sub exited via a loop exit op (such as last) (#16608)

fixed a bug where a regexp warning could leak memory.

prevent a confusing undefined warning message when accessing a sub parameter that was placeholder for a hash element indexed by an undef key (#22423)

What Tony has highlighted are the kinds of bug fixes which collectively help to ensure that Perl remains stable, secure and reliable for the many organisations and individuals who depend on it.

With organizations like Geizhals Preisvergleich funding the work which Tony and others put into maintaining the Perl 5 core, we can work together to ensure that the Perl core continues to receive the maintenance which it deserves, for many years to come. Whether you’re a startup using Perl for rapid prototyping or an enterprise running mission-critical systems, your support helps ensure Perl remains reliable for everyone. Please join us on this journey.

For more information on how to become a sponsor, please contact: olaf@perlfoundation.org

Announcing Dancer2 2.0.0

Mon, 15 Sep 2025 18:00:00 +0000

Your Favorite Perl Web Framework, Now Even Better

The Dancer Core Team project is proud to announce the release of Dancer2 2.0.0!

This release has been a long time coming, and while open source sometimes takes longer than we’d like, we believe the wait has been worth it. With fresh documentation, architectural improvements, and developer-friendly new features, version 2.0.0 represents a significant evolution of Dancer2 and the Perl web ecosystem.

If you’d like a more extensive overview, I gave a talk at the Perl and Raku Conference 2025, about Dancer2 2.0.0, covering new features and where we’re headed next. You can watch the full presentation here: Dancer2 2.0.ohh myyy on YouTube →

Release Highlights

Every major release is an opportunity to take stock of where a project stands and where it’s going. For Dancer2, 2.0.0 represents:

A renewed commitment to documentation and developer experience
A modernization of the framework’s core
A better foundation for web development with Perl

Here are some of the most important changes included in Dancer2 2.0.0:

Brand New Documentation Thanks to a grant from the Perl and Raku Foundation, Dancer2’s documentation has been completely rewritten. Clearer guides and reference materials will make it easier than ever to get started and to master advanced features. Read the docs →
Extendable Configuration System Thanks to first-time contributor Mikko Koivunalho, the new configuration system allows for greater flexibility in how Dancer2 applications are configured and extended. Developers can build new configuration modules, and even integrate configuration systems from other applications or frameworks.

Building on this work, long-time core team member Yanick Champoux enhanced the new configuration system even further, enabling additional configuration readers to be bootstrapped by the default one. Learn more in the config guide →
A Leaner Core Distribution Why have two templating systems in the core framework when you can have zero?
- Dancer2::Template::Simple has been removed from the core. It is now available as a separate distribution on MetaCPAN for migration projects from Dancer 1.
- Our fork of Template::Tiny has been retired, with its improvements merged upstream (thanks, Karen Etheridge!), and Dancer2::Template::TemplateTiny is now just an adapter for the official version. See the documentation →
Smarter Data Handling Dancer2 now supports configurable data/secrets censoring using Data::Censor, helping developers protect sensitive information in logs and debug pages. Learn more about Data::Censor →
Better Logging and Debugging Hooks are now logged as they are executed, and a brand-new hook — on_hook_exception — provides a way to handle unexpected issues more gracefully. See the hooks documentation →
CLI Improvements The command-line interface also received some attention:
- Allows new Dancer2 applications to be scaffolded from the Dancer2 Tutorial.
- Behind-the-scenes improvements to allow future scaffolding of plugins and extensions.

Thank You to Our Contributors

Dancer2 is what it is because of its community. A heartfelt thank you goes out to everyone who made this release possible, including:

Mikko Koivunalho, for his first-time contribution of the extendable configuration system.
Yanick Champoux, for building on Mikko’s work and extending what’s possible with the new configuration system.
Karen Etheridge, for integrating improvements back into Template::Tiny, helping us to streamline the Dancer2 core.
The Perl and Raku Foundation, for supporting the documentation grant that gave us our brand-new docs.
Sawyer X, for being a great grant manager, and his endless patience, great suggestions, and encouragement throughout the grant process.
And of course, everyone who tested, reported issues, submitted patches, and offered feedback along the way.

Your contributions and support are what keep the project moving forward. ❤️

What’s Next?

Dancer2 remains an active, community-driven project, and version 2.0.0 shows our continued dedication to advancing the framework and supporting our community.

We invite you to try out Dancer2 2.0.0, explore the new documentation, and join the conversation on GitHub or the project’s mailing list.

Keep Dancing!

We’re excited about this release and can’t wait to see what the Perl community builds with it.

Jason (CromeDome) and the Dancer2 Core Team

My Guilty Perl Obsession

Tue, 26 Aug 2025 11:09:09 +0000

The Guilt

I consider myself successful.
I’m 45, with a sportscar, a house, a family, and a small business now 30 years old.
I made good decisions.
My car is 15-years old, my monitors are 20-years old, my chair is 25-years old, my desk is 25-years old.
My computers historically last 10-years; I just tossed a mouse that survived 8-years.
Perl.
My first venture into server-side web development was Lotus Notes…that lasted 5 days.
Perl has lasted me nearly 30 years, and we sure as hell ain’t done yet!

I didn’t meet the perl community until the Toronto conference in 2023.
That’s where I saw the faces.
That’s when I saw the humanity.
That’s why I felt the guilt.

I’d paid for my car, for my house, for my computers, my desk, and my chair.
Perl came for free.
I didn’t pay for it; I didn’t work for it.
But at the conference, before me stood the people who did.

I’ve worked hard for my success.
It’s clear to me now that I wasn’t the only one working hard for my success.
How much of your success is because of all of their hard work?
And how much have you contributed in return?
Me? I contributed absolutely nothing.

That’s my guilt.
But this is the very best kind of guilt.
Because it’s not too late!
In fact, now is the perfect time.

I’ve hired members of the perl community.
That’s a start.
I’m donating directly to the foundation.
And I intend to continue doing so.
If my success depends on perl, then perl depends on my success.

The Pleasure

Perl was always a perfect fit for me.
As a syntax, it was concise, yet flexible.
My code’s form could mirror its function.
How perfectly splendid.

Perl is the basis of Holophrastic’s web development platform: Pipelines.
As new popular languages have come along, they’re touted as the best new amazing modern.
And then they vanish, supplanted by the very next amazing.
Had I invested in each, I’d have more archives of code than formats of music albums.

Instead, I rest easy in the knowledge that perl will always keep up.
xls, xlsx, pdf, mysql, mariadb, imagemagick, json, curl
There will always be a cpan module waiting for me when I need it.
My clients have no idea the power that I wield with my fingertips.
AI translations, image processing craziness, survey systems built for 100'000 concurrent test-takers…
And while all of that definitely took some expertise on my end,
Perl created exactly zero hurdles,
It never got in the way at all.

Longevity.
Perl’s got it.
The Perl Foundation’s got it.
The Perl Community’s got it.
I’ve got it.
The lines between? All blurry.

Holophrastic Enterprises Inc.

Custom Business Web Development

I work as what I’d call an inside-contractor.
I’m a speed-dial (is that still a thing?) phone call away.
Occasionally, a client will call me a dozen times in a day.
I’m closer than their colleague in the next office.

It usually starts with “a website”
the typical sales- or marketing-oriented something-pretty

Then business takes over.
A product-configuration wizard
A sales-commission calculator
Can you connect to our accounting software and provide the reports that it can’t?
Can you replace our warehouse-production backend?
What about a portal?

Yes; yes I can.
One business obstacle at a time.

Now, 30 years later, I often encounter code & comments, decades old.
The feeling I get from seeing a line that’s now older than the me who wrote it…
I used to feel alone.
I now feel that thanks to the perl community, I was never alone.

Holophrastic is proud to be the first sponsor of the 2026 Perl and Raku Conference.

SUSE Donates USD 11,500 to The Perl and Raku Foundation

Thu, 07 Aug 2025 09:55:26 +0000

The Perl and Raku Foundation (TPRF) is thrilled to announce a substantial $11,500 donation from SUSE, one of the world’s leading enterprise Linux and cloud-native and AI solutions providers. This generous contribution bolsters the Perl 5 Core Maintenance Fund and demonstrates SUSE’s commitment to the open-source ecosystem.

This donation from SUSE is actually made up of two parts. $10,000 is being donated by SUSE LLC and an additional $1,500 is being provided by The SUSE Open Source Network, to support the development and sustainability of Perl. This aligns with the network’s mission to empower and support open source communities.

Perl is a Fundamental Component of the SUSE Ecosystem

“At SUSE, Perl is a fundamental component and member of our ecosystem,” explains Miguel Pérez Colino, Director of Operations, Linux Product Management & Marketing. “We provide it as part of our Linux offerings by actively supporting Perl packages in SUSE Linux Enterprise and openSUSE. We use it extensively in our toolset, powering among others OpenQA and Open Build Service, this last one is used to build not just Linux packages but also Kubernetes.”

The Perl-Powered OpenQA and Open Build Service

SUSE’s OpenQA project is an automated testing framework that ensures quality across countless hardware configurations and software combinations. At its heart is Perl, orchestrating complex test scenarios with the reliability that system administrators have come to expect.

Similarly, Open Build Service is running on many services written in Perl. represents the modern evolution of package management, creating not just traditional Linux packages but also container images and Kubernetes distributions.

Sustaining the Digital Commons

SUSE’s donation is a demonstration of digital stewardship—the recognition that the tools we rely upon require active investment to remain secure, efficient, and relevant.

“We are proudly donating to The Perl and Raku Foundation (TPRF) to ensure Perl’s continued development and health, which is vital to the open-source world, we are part of, and we champion,” Colino continues.

This investment addresses some critical aspects of language maintenance:

Security Vigilance: In an era of increasing cyber threats, timely security patches aren’t optional—they’re essential. TPRF’s maintenance fund ensures that vulnerabilities can be addressed promptly, protecting countless systems worldwide.

Performance Evolution: Modern computing demands continue to evolve. The fund supports ongoing optimisation efforts that keep Perl competitive in today’s performance-conscious environment.

Platform Diversity: As computing platforms proliferate—from traditional servers to edge devices to cloud containers—Perl must remain compatible and efficient across this expanding landscape.

Community Responsiveness: Bug reports and feature requests from the global Perl community require careful evaluation and implementation. This fund ensures these contributions don’t languish unaddressed.

A Partnership Model for Open Source Sustainability

SUSE’s contribution represents more than financial support—it’s a blueprint for sustainable open-source stewardship. When organisations that build upon open-source foundations reinvest in those foundations, they create a virtuous cycle that benefits everyone. It’s a recognition that the digital commons we all depend upon flourish only through collective stewardship.

MetaCPAN's Traffic Crisis: An Eventual Success Story

Mon, 28 Jul 2025 22:01:46 +0000

"Amelia's Sad Face" by donnierayjones is licensed under CC BY 2.0 .

MetaCPAN.org, the essential search engine for Perl’s CPAN repository, has faced months of severe traffic issues that brought the service to its knees with frequent 503 errors. Here’s how the team fought back against an army of misbehaving bots and hostile traffic.

The Problem Emerges

MetaCPAN began experiencing multiple 503 service errors daily, disrupting access for legitimate Perl developers worldwide. Traditional monitoring failed to identify the root cause of traffic spikes overwhelming the infrastructure.

Initial Investigation Phase

The team implemented basic monitoring and took preliminary defensive measures:

Deployed uWSGI stats monitoring tools to track application performance
Updated robots.txt to explicitly list bots and specify crawling restrictions
Began manual IP blocking of obvious bad actors
Attempted to deploy Anubis rate limiting (ultimately failed and was rolled back)

The Datadog Breakthrough

Partnership with Datadog transformed visibility into the problem:

Established comprehensive logging pipeline sending Fastly CDN logs for both web and API services to Datadog
Deployed Kubernetes Datadog agent to cluster
Created public dashboard showing real-time traffic metrics
Built private dashboard specifically to identify problematic IPs and user agents

Result: Finally able to see the enemy—specific IP ranges (particularly from Alibaba.com) and user agents generating massive request volumes. However, manual blocking proved unsustainable, requiring constant vigilance and rapid response.

Escalating Defences

The team implemented more sophisticated blocking:

Deployed VCL snippets in Fastly to block based on user agents (later replaced with Next Gen WAF)
Blocked extensive IP ranges using Fastly’s IP Block list feature
Implemented additional request rate limiting
Partnered with Fastly for free enterprise services including DDoS protection

Limitation: Manual processes couldn’t keep pace with evolving attack patterns.

Next-Generation WAF Implementation

Deployment of Fastly’s Web Application and API Protection:

Enabled next-gen WAF to automatically identify and block suspect bots
Implemented categorical blocking for known bad traffic types
Reduced manual intervention requirements significantly

Progress: Noticeable improvement, but sophisticated attacks still overwhelmed the service during peak periods.

The Dynamic Challenge Solution

Final defensive layer was activated:

Deployed Fastly’s Dynamic Challenge WAF feature
Intelligent challenge system filtered automated bots whilst allowing legitimate users through
Dramatic reduction in successful attacks reaching MetaCPAN infrastructure

Current State: Victory Through Data

Today’s public Datadog dashboard tells the success story in real-time metrics:

In the last week the number of requests handled broke down as follows:

5,190,000 bad bot requests (this includes AI scrapers) blocked
3,290,000 challenges issued
579,000 requests rate limited
1,720,000 legitimate requests served (much of this from Fastly’s CDN cache), with the remainder reaching the origin servers and being successfully served to end users.

So about 80% of all traffic is now blocked.

The numbers demonstrate the scale of the threat MetaCPAN faced and the effectiveness of the layered defence strategy.

We have RSS feeds and a dedicated API which can be easily accessed through MetaCPAN::Client for anyone who wants to get data from us without scraping the site. We do ask that people register their user agent.

Community Heroes

This infrastructure battle was won through generous community support:

Break down of steps can be found on ticket https://github.com/metacpan/metacpan-k8s/issues/154

Fastly and Datadog deserve particular recognition for donating enterprise-grade services. Without these contributions, MetaCPAN couldn’t operate at the required scale and reliability.

Additional sponsors listed at https://metacpan.org/about/sponsors continue supporting this vital community resource, though operational costs remain significant.

How to help: The Perl community can support MetaCPAN’s ongoing operations through https://opencollective.com/metacpan-core, ensuring this essential service remains available for all developers.

Analysing FIT data with Perl: interactive data analysis

Sun, 27 Jul 2025 11:46:04 +0000

Printing statistics to the terminal or plotting data extracted from FIT files is all well and good. One problem is that the feedback loops are long. Sometimes questions are better answered by playing with the data directly. Enter the Perl Data Language.

Interactive data analysis

For more fine-grained analysis of our FIT file data, it’d be great to be able to investigate it interactively. Other languages such as Ruby, Raku and Python have a built-in REPL.¹ Yet Perl doesn’t.² But help is at hand! PDL (the Perl Data Language) is designed to be used interactively and thus has a REPL we can use to manipulate and investigate our activity data.³

Getting set up

Before we can use PDL, we’ll have to install it:

$ cpanm PDL

After it has finished installing (this can take a while), you’ll be able to start the perlDL shell with the pdl command:

perlDL shell v1.357
 PDL comes with ABSOLUTELY NO WARRANTY. For details, see the file
 'COPYING' in the PDL distribution. This is free software and you
 are welcome to redistribute it under certain conditions, see
 the same file for details.
ReadLines, NiceSlice, MultiLines  enabled
Reading PDL/default.perldlrc...
Found docs database /home/vagrant/perl5/perlbrew/perls/perl-5.38.3/lib/site_perl/5.38.3/x86_64-linux/PDL/pdldoc.db
Type 'help' for online help
Type 'demo' for online demos
Loaded PDL v2.100 (supports bad values)

Note: AutoLoader not enabled ('use PDL::AutoLoader' recommended)

pdl>

To exit the pdl shell, enter Ctrl-D at the prompt and you’ll be returned to your terminal.

Cleaning up to continue

To manipulate the data in the pdl shell, we want to be able to call individual routines from the geo-fit-plot-data.pl script. This way we can use the arrays that some of the routines return to initialise PDL data objects.

It’s easier to manipulate the data if we get ourselves a bit more organised first.⁴ In other words, we need to extract the routines into a module, which will make calling the code we created earlier from within pdl much easier.

Before we create a module, we need to do some refactoring. One thing that’s been bothering me is the way the plot_activity_data() subroutine also parses and manipulates date/time data. This routine should be focused on plotting data, not on massaging its requirements into the correct form. Munging the date/time data is something that should happen in its own routine. This way we encapsulate the concepts and abstract away the details. Another way of saying this is that the plotting routine shouldn’t “know” how to manipulate date/time information to do its job.

To this end, I’ve moved the time extraction code into a routine called get_time_data():

sub get_time_data {
    my @activity_data = @_;

    # get the epoch time for the first point in the time data
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;

    # convert timestamp data to elapsed minutes from start of activity
    my @times = map {
        my $dt = $date_parser->parse_datetime($_);
        my $epoch_time = $dt->epoch;
        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
        $elapsed_time;
    } @timestamps;

    return @times;
}

The main change here in comparison to the previous version of the code is that we pass the activity data as an argument to get_time_data(), returning the @times array to the calling code.

The code creating the date string used in the plot title now also resides in its own function:

sub get_date {
    my @activity_data = @_;

    # determine date from timestamp data
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my $dt = $date_parser->parse_datetime($timestamps[0]);
    my $date = $dt->strftime("%Y-%m-%d");

    return $date;
}

Where again, we’re passing the @activity_data array to the function. It then returns the $date string that we use in the plot title.

Both of these routines use the $date_parser object, which I’ve extracted into a constant in the main script scope:

my $date_parser = DateTime::Format::Strptime->new(
    pattern => "%Y-%m-%dT%H:%M:%SZ",
    time_zone => 'UTC',
);

Making a mini-module

It’s time to make our module. I’m not going to create the full Perl module infrastructure here, as it’s not necessary for our current goal. I want to import a module called Geo::FIT::Utils and then access the functions that it provides.⁵ Thus–in an appropriate project directory–we need to create a file called lib/Geo/FIT/Utils.pm as well as its associated path:

$ mkdir -p lib/Geo/FIT
$ touch lib/Geo/FIT/Utils.pm

Opening the file in an editor and entering this stub module code:

 1package Geo::FIT::Utils;
 2
 3use Exporter 5.57 'import';
 4
 5
 6our @EXPORT_OK = qw(
 7    extract_activity_data
 8    show_activity_statistics
 9    plot_activity_data
10    get_time_data
11    num_parts
12);
13
141;

we now have the scaffolding of a module that (at least, theoretically) exports the functionality we need.

Line 1 specifies the name of the module. Note that the module’s name must match its path on the filesystem, hence why we created the file Geo/FIT/Utils.pm.

We import the Exporter module (line 3) so that we can specify the functions to export. This is the @EXPORT_OK array’s purpose (lines 6-12).

Finally, we end the file on line 14 with the code 1;. This line is necessary so that importing the package (which in this case is also a module) returns a true value. The value 1 is synonymous with Boolean true in Perl, hence why it’s best practice to end module files with 1;.

Copying all the code except the main() routine from geo-fit-plot-data.pl into Utils.pm, we end up with this:

package Geo::FIT::Utils;

use strict;
use warnings;

use Exporter 5.57 'import';
use Geo::FIT;
use Scalar::Util qw(reftype);
use List::Util qw(max sum);
use Chart::Gnuplot;
use DateTime::Format::Strptime;


my $date_parser = DateTime::Format::Strptime->new(
    pattern => "%Y-%m-%dT%H:%M:%SZ",
    time_zone => 'UTC',
);

sub extract_activity_data {
    my $fit = Geo::FIT->new();
    $fit->file( "2025-05-08-07-58-33.fit" );
    $fit->open or die $fit->error;

    my $record_callback = sub {
        my ($self, $descriptor, $values) = @_;
        my @all_field_names = $self->fields_list($descriptor);

        my %event_data;
        for my $field_name (@all_field_names) {
            my $field_value = $self->field_value($field_name, $descriptor, $values);
            if ($field_value =~ /[a-zA-Z]/) {
                $event_data{$field_name} = $field_value;
            }
        }

        return \%event_data;
    };

    $fit->data_message_callback_by_name('record', $record_callback ) or die $fit->error;

    my @header_things = $fit->fetch_header;

    my $event_data;
    my @activity_data;
    do {
        $event_data = $fit->fetch;
        my $reftype = reftype $event_data;
        if (defined $reftype && $reftype eq 'HASH' && defined %$event_data{'timestamp'}) {
            push @activity_data, $event_data;
        }
    } while ( $event_data );

    $fit->close;

    return @activity_data;
}

# extract and return the numerical parts of an array of FIT data values
sub num_parts {
    my $field_name = shift;
    my @activity_data = @_;

    return map { (split ' ', $_->{$field_name})[0] } @activity_data;
}

# return the average of an array of numbers
sub avg {
    my @array = @_;

    return (sum @array) / (scalar @array);
}

sub show_activity_statistics {
    my @activity_data = @_;

    print "Found ", scalar @activity_data, " entries in FIT file\n";
    my $available_fields = join ", ", sort keys %{$activity_data[0]};
    print "Available fields: $available_fields\n";

    my $total_distance_m = (split ' ', ${$activity_data[-1]}{'distance'})[0];
    my $total_distance = $total_distance_m/1000;
    print "Total distance: $total_distance km\n";

    my @speeds = num_parts('speed', @activity_data);
    my $maximum_speed = max @speeds;
    my $maximum_speed_km = $maximum_speed*3.6;
    print "Maximum speed: $maximum_speed m/s = $maximum_speed_km km/h\n";

    my $average_speed = avg(@speeds);
    my $average_speed_km = sprintf("%0.2f", $average_speed*3.6);
    $average_speed = sprintf("%0.2f", $average_speed);
    print "Average speed: $average_speed m/s = $average_speed_km km/h\n";

    my @powers = num_parts('power', @activity_data);
    my $maximum_power = max @powers;
    print "Maximum power: $maximum_power W\n";

    my $average_power = avg(@powers);
    $average_power = sprintf("%0.2f", $average_power);
    print "Average power: $average_power W\n";

    my @heart_rates = num_parts('heart_rate', @activity_data);
    my $maximum_heart_rate = max @heart_rates;
    print "Maximum heart rate: $maximum_heart_rate bpm\n";

    my $average_heart_rate = avg(@heart_rates);
    $average_heart_rate = sprintf("%0.2f", $average_heart_rate);
    print "Average heart rate: $average_heart_rate bpm\n";
}

sub plot_activity_data {
    my @activity_data = @_;

    # extract data to plot from full activity data
    my @times = get_time_data(@activity_data);
    my @heart_rates = num_parts('heart_rate', @activity_data);
    my @powers = num_parts('power', @activity_data);

    # plot data
    my $date = get_date(@activity_data);
    my $chart = Chart::Gnuplot->new(
        output => "watopia-figure-8-heart-rate-and-power.png",
        title  => "Figure 8 in Watopia on $date: heart rate and power over time",
        xlabel => "Elapsed time (min)",
        ylabel => "Heart rate (bpm)",
        terminal => "png size 1024, 768",
        xtics => {
            incr => 5,
        },
        ytics => {
            mirror => "off",
        },
        y2label => 'Power (W)',
        y2range => [0, 1100],
        y2tics => {
            incr => 100,
        },
    );

    my $heart_rate_ds = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@heart_rates,
        style => "lines",
    );

    my $power_ds = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@powers,
        style => "lines",
        axes => "x1y2",
    );

    $chart->plot2d($power_ds, $heart_rate_ds);
}

sub get_time_data {
    my @activity_data = @_;

    # get the epoch time for the first point in the time data
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;

    # convert timestamp data to elapsed minutes from start of activity
    my @times = map {
        my $dt = $date_parser->parse_datetime($_);
        my $epoch_time = $dt->epoch;
        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
        $elapsed_time;
    } @timestamps;

    return @times;
}

sub get_date {
    my @activity_data = @_;

    # determine date from timestamp data
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my $dt = $date_parser->parse_datetime($timestamps[0]);
    my $date = $dt->strftime("%Y-%m-%d");

    return $date;
}

our @EXPORT_OK = qw(
    extract_activity_data
    show_activity_statistics
    plot_activity_data
    get_time_data
    num_parts
);

1;

… which is what we had before, but put into a nice package for easier use.

One upside to having put all this code into a module is that the geo-fit-plot-data.pl script is now much simpler:

use strict;
use warnings;

use Geo::FIT::Utils qw(
    extract_activity_data
    show_activity_statistics
    plot_activity_data
);


sub main {
    my @activity_data = extract_activity_data();

    show_activity_statistics(@activity_data);
    plot_activity_data(@activity_data);
}

main();

Poking and prodding

We’re now ready to investigate our power and heart rate data interactively!

Start pdl and enter use lib 'lib' at the pdl> prompt so that it can find our new module:⁶

$ pdl
perlDL shell v1.357
 PDL comes with ABSOLUTELY NO WARRANTY. For details, see the file
 'COPYING' in the PDL distribution. This is free software and you
 are welcome to redistribute it under certain conditions, see
 the same file for details.
ReadLines, NiceSlice, MultiLines  enabled
Reading PDL/default.perldlrc...
Found docs database /home/vagrant/perl5/perlbrew/perls/perl-5.38.3/lib/site_perl/5.38.3/x86_64-linux/PDL/pdldoc.db
Type 'help' for online help
Type 'demo' for online demos
Loaded PDL v2.100 (supports bad values)

Note: AutoLoader not enabled ('use PDL::AutoLoader' recommended)

pdl> use lib 'lib'

Now import the functions we wish to use:

pdl> use Geo::FIT::Utils qw(extract_activity_data get_time_data num_parts)

Since we need the activity data from the FIT file to pass to the other routines, we grab it and put it into a variable:

pdl> @activity_data = extract_activity_data

We also need to load the time data:

pdl> @times = get_time_data(@activity_data)

which we can then read into a PDL array:

pdl> $time = pdl \@times

With the time data in a PDL array, we can manipulate it more easily. For instance, we can display elements of the array with the PDL print statement in combination with the splice() method. The following code shows the last five elements of the $time array:

pdl> print $time->slice("-1:-5")
[54.5333333333333 54.5166666666667 54.5 54.4833333333333 54.4666666666667]

Loading power output and heart rate data into PDL arrays works similarly:

pdl> @powers = num_parts('power', @activity_data)

pdl> $power = pdl \@powers

pdl> @heart_rates = num_parts('heart_rate', @activity_data)

pdl> $heart_rate = pdl \@heart_rates

In the previous article, we wanted to know what the maximum power was for the second sprint. Here’s the graph again for context:

Eyeballing the graph from above, we can see that the second sprint occurred between approximately 47 and 48 minutes elapsed time. We know that the arrays of time and power data all have the same length. Thus, if we find out the indices of the $time array between these times, we can use them to select the corresponding power data. To get array indices for known data values, we use the PDL which command:

pdl> $indices = which(47 < $time & $time < 48)

pdl> print $indices
[2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835
 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850
 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865
 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879]

We can check that we’ve got the correct range of time values by passing the $indices array as a slice of the $time array:

pdl> print $time($indices)
[47.0166666666667 47.0333333333333 47.05 47.0666666666667 47.0833333333333
 47.1 47.1166666666667 47.1333333333333 47.15 47.1666666666667
 47.1833333333333 47.2 47.2166666666667 47.2333333333333 47.25
 47.2666666666667 47.2833333333333 47.3 47.3166666666667 47.3333333333333
 47.35 47.3666666666667 47.3833333333333 47.4 47.4166666666667
 47.4333333333333 47.45 47.4666666666667 47.4833333333333 47.5
 47.5166666666667 47.5333333333333 47.55 47.5666666666667 47.5833333333333
 47.6 47.6166666666667 47.6333333333333 47.65 47.6666666666667
 47.6833333333333 47.7 47.7166666666667 47.7333333333333 47.75
 47.7666666666667 47.7833333333333 47.8 47.8166666666667 47.8333333333333
 47.85 47.8666666666667 47.8833333333333 47.9 47.9166666666667
 47.9333333333333 47.95 47.9666666666667 47.9833333333333]

The time values lie between 47 and 48, so we can conclude that we’ve selected the correct indices.

Note that we have to use the bitwise logical AND operator here because it operates on an element-by-element basis across the array.

Selecting $power array values at these indices is as simple as passing the $indices array as a slice:

pdl> print $power($indices)
[229 231 232 218 210 204 255 252 286 241 231 237 260 256 287 299 318 337 305
 276 320 289 280 301 320 303 395 266 302 341 299 287 309 279 294 284 266 281
 367 497 578 512 762 932 907 809 821 847 789 740 657 649 722 715 669 657 705
 643 647]

Using the max() method on this output gives us the maximum power:

pdl> print $power($indices)->max
932

In other words, the maximum power for the second sprint was 932 W. Not as good as the first sprint (which achieved 1023 W), but I was getting tired by this stage.

The same procedure allows us to find the maximum power for the first sprint with PDL. Again, eyeballing the graph above, we can see that the peak for the first sprint occurred between 24 and 26 minutes. Constructing the query in PDL, we have

pdl> print $power(which(24 < $time & $time < 26))->max
1023

which gives the maximum power value we expect.

We can also find out the maximum heart rate values around these times. E.g. for the first sprint:

pdl> print $heart_rate(which(24 < $time & $time < 26))->max
157

in other words, 157 bpm. For the second sprint, we have:

pdl> print $heart_rate(which(47 < $time & $time < 49))->max
165

i.e. 165 bpm, which matches the value that we found earlier. Note that I broadened the range of times to search over heart rate data here because its peak occurred a bit after the power peak for the second sprint.

Looking forward

Where to from here? Well, we could extend this code to handle processing multiple FIT files. This would allow us to find trends over many activities and longer periods. Perhaps there are other data sources that one could combine with longer trends. For instance, if one has access to weight data over time, then it’d be possible to work out things like power-to-weight ratios. Maybe looking at power and heart rate trends over a longer time can identify things such as overtraining. I’m not a sport scientist, so I don’t know how to go about that, yet it’s a possibility. Since we’ve got fine-grained, per-ride data, if we can combine this with longer-term analysis, there are probably many more interesting tidbits hiding in there that we can look at and think about.

Open question

One thing I haven’t been able to work out is where the calorie information is. As far as I can tell, Zwift calculates how many calories were burned during a given ride. Also, if one uploads the FIT file to a service such as Strava, then it too shows calories burned and the value is the same. This would imply that Strava is only displaying a value stored in the FIT file. So where is the calorie value in the FIT data? I’ve not been able to find it in the data messages that Geo::FIT reads, so I’ve no idea what’s going on there.

Conclusion

What have we learned? We’ve found out how to read, analyse and plot data from Garmin FIT files all by using Perl modules. Also, we’ve learned how to investigate the data interactively by using the PDL shell. Cool!

One main takeaway that might not be obvious is that you don’t really need online services such as Strava. You should now have the tools to process, analyse and visualise data from your own FIT files. With Geo::FIT, Chart::Gnuplot and a bit of programming, you can glue together the components to provide much of the same (and in some cases, more) functionality yourself.

I wish you lots of fun when playing around with FIT data!

REPL stands for read-eval-print loop and is an environment where one can interactively enter programming language commands and manipulate data. ↩︎
It is, however, possible to (ab)use the Perl debugger and use it as a kind of REPL. Enter perl -de0 and you’re in a Perl environment much like REPLs in other languages. ↩︎
Many thanks to Harald Jörg for pointing this out to me at the recent German Perl and Raku Workshop. ↩︎
This is an application of “first make the change easy, then make the easy change” (paraphrasing Kent Beck). An important point often overlooked in this quote is that making the change easy can be hard. ↩︎
Not a particularly imaginative name, I know. ↩︎
The documented way to add a path to @INC in pdl is via the -Ilib command line option. Unfortunately, this didn’t work in my test environment: the local lib/ path wasn’t added to @INC and hence using the Geo::FIT::Utils module failed with the error that it couldn’t be located. ↩︎

Proxmox Donates €10,000 to The Perl and Raku Foundation

Wed, 23 Jul 2025 10:24:25 +0000

The Perl and Raku Foundation (TPRF) is delighted to announce a generous €10,000 donation from Proxmox Server Solutions GmbH, supporting the critical Perl 5 Core Maintenance Fund. Corporate partnerships play a critical role in enabling TPRF to fulfill its mission.

A Partner in Open Source

Proxmox Virtual Environment is a complete, open-source server management platform for enterprise virtualization. It tightly integrates the KVM hypervisor and Linux Containers (LXC), software-defined storage and networking functionality, on a single platform.

Proxmox is an example of an open source company that has built enterprise-grade virtualization technology while maintaining transparency, community engagement, and accessibility as core principles.

Sustaining a Foundation of Modern Computing

The Perl programming language remains a cornerstone of system administration, bioinformatics, web development, and countless other critical applications across industries. However, the ongoing maintenance and development of Perl’s core requires dedicated funding to ensure continued stability, security updates, and feature enhancements.

TPRF is dedicated to the advancement of the Perl and Raku programming languages, through open discussion, collaboration, design, and code. This mission extends beyond language development to encompass community building, educational initiatives, and the crucial task of maintaining the robust infrastructure that millions of applications depend upon.

The Critical Nature of Core Maintenance

Without sustained funding for core maintenance work, even the most established programming languages risk stagnation or security vulnerabilities. The Perl 5 Core Maintenance Fund specifically addresses:

Security Updates: Ensuring timely patches for discovered vulnerabilities
Performance Optimizations: Maintaining competitive execution speeds
Platform Compatibility: Supporting new operating systems and architectures
Bug Resolution: Addressing issues reported by the global Perl community
Documentation Maintenance: Keeping comprehensive guides current and accessible

Proxmox’s contribution directly enables this essential work to continue uninterrupted, demonstrating a forward-thinking approach to technology stewardship.

A Shared Vision for Open Source Sustainability

This donation reflects a broader understanding within the technology industry that sustainable open source ecosystems require active investment from organizations that benefit from these tools. As companies increasingly seek cost-effective alternatives to proprietary solutions, the importance of maintaining robust open source alternatives becomes paramount.

Looking Forward

By investing in Perl’s continued development, Proxmox contributes to a programming language ecosystem that serves developers, system administrators, and organizations worldwide.

This partnership comes at a crucial time for TPRF. Community support from sponsors like Proxmox enables the foundation to maintain its diverse portfolio of community-serving initiatives.

Analysing FIT data with Perl: producing PNG plots

Wed, 25 Jun 2025 10:10:08 +0000

Last time, we worked out how to extract, collate, and print statistics about the data contained in a FIT file. Now we’re going to take the next logical step and plot the time series data.

Start plotting with Gnu

Now that we’ve extracted data from the FIT file, what else can we do with it? Since this is time series data, the most natural next step is to visualise the data values over time. Since I know that Gnuplot handles time series data well,¹ I chose to use Chart::Gnuplot to plot the data.

An additional point in Gnuplot’s favour is that it can plot two datasets on the same graph, each with its own y-axis. Such functionality is handy when searching for correlations between datasets of different y-axis scales and ranges that share the same baseline data series.

Clearly Chart::Gnuplot relies on Gnuplot, so we need to install it first:

$ sudo apt install -y gnuplot

Now we can install Chart::Gnuplot with cpanm:

$ cpanm Chart::Gnuplot

Putting our finger on the pulse

Something I like looking at is how my heart rate evolved throughout a ride; it gives me an idea of how much effort I was putting in. So, we’ll start off by looking at how the heart rate data varied over time. In other words, we want time on the x-axis and heart rate on the y-axis.

One great thing about Gnuplot is that if you give it a format string for the time data, then plotting “just works”. In other words, explicit conversion to datetime data for the x-axis is unnecessary.

Here’s a script to extract the FIT data from our example data file. It displays some statistics about the activity and plots heart rate versus time. I’ve given the script the filename geo-fit-plot-data.pl:

  1use strict;
  2use warnings;
  3
  4use Geo::FIT;
  5use Scalar::Util qw(reftype);
  6use List::Util qw(max sum);
  7use Chart::Gnuplot;
  8
  9
 10sub main {
 11    my @activity_data = extract_activity_data();
 12
 13    show_activity_statistics(@activity_data);
 14    plot_activity_data(@activity_data);
 15}
 16
 17sub extract_activity_data {
 18    my $fit = Geo::FIT->new();
 19    $fit->file( "2025-05-08-07-58-33.fit" );
 20    $fit->open or die $fit->error;
 21
 22    my $record_callback = sub {
 23        my ($self, $descriptor, $values) = @_;
 24        my @all_field_names = $self->fields_list($descriptor);
 25
 26        my %event_data;
 27        for my $field_name (@all_field_names) {
 28            my $field_value = $self->field_value($field_name, $descriptor, $values);
 29            if ($field_value =~ /[a-zA-Z]/) {
 30                $event_data{$field_name} = $field_value;
 31            }
 32        }
 33
 34        return \%event_data;
 35    };
 36
 37    $fit->data_message_callback_by_name('record', $record_callback ) or die $fit->error;
 38
 39    my @header_things = $fit->fetch_header;
 40
 41    my $event_data;
 42    my @activity_data;
 43    do {
 44        $event_data = $fit->fetch;
 45        my $reftype = reftype $event_data;
 46        if (defined $reftype && $reftype eq 'HASH' && defined %$event_data{'timestamp'}) {
 47            push @activity_data, $event_data;
 48        }
 49    } while ( $event_data );
 50
 51    $fit->close;
 52
 53    return @activity_data;
 54}
 55
 56# extract and return the numerical parts of an array of FIT data values
 57sub num_parts {
 58    my $field_name = shift;
 59    my @activity_data = @_;
 60
 61    return map { (split ' ', $_->{$field_name})[0] } @activity_data;
 62}
 63
 64# return the average of an array of numbers
 65sub avg {
 66    my @array = @_;
 67
 68    return (sum @array) / (scalar @array);
 69}
 70
 71sub show_activity_statistics {
 72    my @activity_data = @_;
 73
 74    print "Found ", scalar @activity_data, " entries in FIT file\n";
 75    my $available_fields = join ", ", sort keys %{$activity_data[0]};
 76    print "Available fields: $available_fields\n";
 77
 78    my $total_distance_m = (split ' ', ${$activity_data[-1]}{'distance'})[0];
 79    my $total_distance = $total_distance_m/1000;
 80    print "Total distance: $total_distance km\n";
 81
 82    my @speeds = num_parts('speed', @activity_data);
 83    my $maximum_speed = max @speeds;
 84    my $maximum_speed_km = $maximum_speed*3.6;
 85    print "Maximum speed: $maximum_speed m/s = $maximum_speed_km km/h\n";
 86
 87    my $average_speed = avg(@speeds);
 88    my $average_speed_km = sprintf("%0.2f", $average_speed*3.6);
 89    $average_speed = sprintf("%0.2f", $average_speed);
 90    print "Average speed: $average_speed m/s = $average_speed_km km/h\n";
 91
 92    my @powers = num_parts('power', @activity_data);
 93    my $maximum_power = max @powers;
 94    print "Maximum power: $maximum_power W\n";
 95
 96    my $average_power = avg(@powers);
 97    $average_power = sprintf("%0.2f", $average_power);
 98    print "Average power: $average_power W\n";
 99
100    my @heart_rates = num_parts('heart_rate', @activity_data);
101    my $maximum_heart_rate = max @heart_rates;
102    print "Maximum heart rate: $maximum_heart_rate bpm\n";
103
104    my $average_heart_rate = avg(@heart_rates);
105    $average_heart_rate = sprintf("%0.2f", $average_heart_rate);
106    print "Average heart rate: $average_heart_rate bpm\n";
107}
108
109sub plot_activity_data {
110    my @activity_data = @_;
111
112    my @heart_rates = num_parts('heart_rate', @activity_data);
113    my @times = map { $_->{'timestamp'} } @activity_data;
114
115    my $date = "2025-05-08";
116
117    my $chart = Chart::Gnuplot->new(
118        output => "watopia-figure-8-heart-rate.png",
119        title  => "Figure 8 in Watopia on $date: heart rate over time",
120        xlabel => "Time",
121        ylabel => "Heart rate (bpm)",
122        terminal => "png size 1024, 768",
123        timeaxis => "x",
124        xtics => {
125            labelfmt => '%H:%M',
126        },
127    );
128
129    my $data_set = Chart::Gnuplot::DataSet->new(
130        xdata => \@times,
131        ydata => \@heart_rates,
132        timefmt => "%Y-%m-%dT%H:%M:%SZ",
133        style => "lines",
134    );
135
136    $chart->plot2d($data_set);
137}
138
139main();

A lot has happened between this code and the previous scripts. Let’s review it to see what’s changed.

The biggest changes were structural. I’ve moved the code into separate routines, improving encapsulation and making each more focused on one task.

The FIT file data extraction code I’ve put into its own routine (extract_activity_data(); lines 17-54). This sub returns the array of event data that we’ve been using.

I’ve also created two utility routines num_parts() (lines 57-62) and avg() (lines 65-69). These return the numerical parts of the activity data and average data series value, respectively.

The ride statistics calculation and display code is now located in the show_activity_statistics() routine. Now it’s out of the way, allowing us to concentrate on other things.

The plotting code is new and sits in a sub called plot_activity_data() (lines 109-137). We’ll focus much more on that later.

These routines are called from a main() routine (lines 10-15) giving us a nice bird’s eye view of what the script is trying to achieve. Running all the code is now as simple as calling main() (line 139).

Particulars of pulse plotting

Let’s zoom in on the plotting code in plot_activity_data(). After having imported Chart::Gnuplot at the top of the file (line 7), we need to do a bit of organising before we can set up the chart. We extract the activity data with extract_activity_data() (line 11) and pass this as an argument to plot_activity_data() (line 14). At the top of plot_activity_data() we fetch an array of the numerical heart rate data (line 112) and an array of all the timestamps (line 113).

The activity’s date (line 115) is assigned as a string variable because I want this to appear in the chart’s title. Although the date is present in the activity data, I’ve chosen not to calculate its value until later. This way we get the plotting code up and running sooner, as there’s still a lot to discuss.

Now we’re ready to set up the chart, which takes place on lines 117-127. We create a new Chart::Gnuplot object on line 117 and configure the plot with various keyword arguments to the object’s constructor.

The parameters are as follows:

output specifies the name of the output file as a string. The name I’ve chosen reflects the activity as well as the data being plotted.
title is a string to use as the plot’s title. To provide context, I mention the name of the route (Figure 8) within Zwift’s main virtual world (Watopia) as well as the date of the activity. To highlight that we’re plotting heart rate over time, I’ve mentioned this in the title also.
xlabel is a string describing the x-axis data.
ylabel is a string describing the y-axis data.
the terminal option tells Gnuplot to use the PNG² “terminal”³ and to set its dimensions to 1024x768.
timeaxis tells Gnuplot which axis contains time-based data (in this case the x-axis). This enables Gnuplot to space out the data along the axis evenly. Often, the spacing between points in time-based data isn’t regular; for instance, data points can be missing. Hence, naively plotting unevenly-spaced time data can produce a distorted graph. Telling Gnuplot that the x-axis contains time-based data allows it to add appropriate space where necessary.
xtics is a hash of options to configure the behaviour of the ticks on the x-axis. The setting here displays hour and minute information at each tick mark for our time data. We omit the year, month and day information as this is the same for all data points.

Now that the main chart parameters have been set, we can focus on the data we want to plot. In Chart::Gnuplot parlance, a Chart::Gnuplot::DataSet object represents a set of data to plot. Lines 129-134 instantiate such an object which we later pass to the Chart::Gnuplot object when plotting the data (line 136). One configures Chart::Gnuplot::DataSet objects similarly to how Chart::Gnuplot objects are constructed: by passing various options to its constructor. These options include the data to plot and how this data should be styled on the graph.

The options used here have the following meanings:

xdata is an array reference to the data to use for the x-axis. If this option is omitted, then Gnuplot uses the array indices of the y-data as the x-axis values.
ydata is an array reference to the data to use for the y-axis.
timefmt specifies the format string Gnuplot should use when reading the time data in the xdata array. Timestamps are strings and we need to inform Gnuplot how to parse them into a form useful for x-axis data. Were the x-axis data a numerical data type, this option wouldn’t be necessary.
style is a string specifying the style to use for plotting the data. In this example, we’re plotting the data points as a set of connected lines. Check out the Chart::Gnuplot documentation for a full list of the available style options.

We finally get to plot the data on line 136. The data set gets passed to the Chart::Gnuplot object as the argument to its plot2d() method. As its name suggests, this plots 2D data, i.e. y versus x. Gnuplot can also display 3D data, in which case we’d call plot3d(). When plotting 3D data we’d have to include a z dimension when setting up the data set.

Running this code

$ perl geo-fit-plot-data.pl

generates this plot:

A couple of things are apparent when looking at this graph. It took me a while to get going (my pulse rose steadily over the first ~15 minutes of the ride) and the time is weird (6 am? Me? Lol, no). We’ll try to explain the heart rate behaviour later.

But first, what’s up with the time data? Did I really start riding at 6 o’clock in the morning? I’m not a morning person, so that’s not right. Also, I’m pretty sure my neighbours wouldn’t appreciate me coughing and wheezing at 6 am while trying to punish myself on Zwift. So what’s going on?

For those following carefully, you might have noticed the trailing Z on the timestamp data. This means that the time zone is UTC. Given that this data is from May and I live in Germany, this implies that the local time would have been 8 am. Still rather early for me, but not too early to disturb the neighbours too much.⁴ In other words, we need to fix the time zone to get the time data right.

Getting into the zone

How do we fix the time zone? I’m glad you asked! We need to parse the timestamp into a DateTime object, set the time zone, and then pass the fixed time data to Gnuplot. It turns out that the standard DateTime library doesn’t parse date/time strings. Instead, we need to use DateTime::Format::Strptime. This module parses date/time strings much like the strptime(3) POSIX function does and returns DateTime objects.

Since the module isn’t part of the core Perl distribution, we need to install it:

$ cpanm DateTime::Format::Strptime

Most of the code changes that follow take place only within the plotting routine (plot_activity_data()). So, I’m going to focus on that from now on and won’t create a new script for the new version of the code.

The first thing to do is to import the DateTime::Format::Strptime module:

 use Scalar::Util qw(reftype);
 use List::Util qw(max sum);
 use Chart::Gnuplot;
+use DateTime::Format::Strptime;

Extending plot_activity_data() to set the correct time zone, we get this code:

 1sub plot_activity_data {
 2    my @activity_data = @_;
 3
 4    # extract data to plot from full activity data
 5    my @heart_rates = num_parts('heart_rate', @activity_data);
 6    my @timestamps = map { $_->{'timestamp'} } @activity_data;
 7
 8    # fix time zone in time data
 9    my $date_parser = DateTime::Format::Strptime->new(
10        pattern => "%Y-%m-%dT%H:%M:%SZ",
11        time_zone => 'UTC',
12    );
13
14    my @times = map {
15        my $dt = $date_parser->parse_datetime($_);
16        $dt->set_time_zone('Europe/Berlin');
17        my $time_string = $dt->strftime("%H:%M:%S");
18        $time_string;
19    } @timestamps;
20
21    # determine date from timestamp data
22    my $dt = $date_parser->parse_datetime($timestamps[0]);
23    my $date = $dt->strftime("%Y-%m-%d");
24
25    # plot data
26    my $chart = Chart::Gnuplot->new(
27        output => "watopia-figure-8-heart-rate.png",
28        title  => "Figure 8 in Watopia on $date: heart rate over time",
29        xlabel => "Time",
30        ylabel => "Heart rate (bpm)",
31        terminal => "png size 1024, 768",
32        timeaxis => "x",
33        xtics => {
34            labelfmt => '%H:%M',
35        },
36    );
37
38    my $data_set = Chart::Gnuplot::DataSet->new(
39        xdata => \@times,
40        ydata => \@heart_rates,
41        timefmt => "%H:%M:%S",
42        style => "lines",
43    );
44
45    $chart->plot2d($data_set);
46}

The timestamp data is no longer read straight into the @times array; it’s stored in the @timestamps temporary array (line 6). This change also makes the variable naming a bit more consistent, which is nice.

To parse a timestamp string into a DateTime object, we need to tell DateTime::Format::Strptime how to format the timestamp (lines 8-12). This is the purpose of the pattern argument in the DateTime::Format::Strptime constructor (line 10). You might have noticed that we used the same pattern when telling Gnuplot what format the time data was in. We also specify the time zone (line 11) to ensure that the date/time data is parsed as UTC.

Next, we fix the time zone in all elements of the @timestamps array (lines 14-19). I’ve chosen to do this within a map here. I could extract this code into a well-named routine, but it does the job for now. The map parses the date/time string into a Date::Time object (line 15) and sets the time zone to Europe/Berlin⁵ (line 16). We only need to plot the time data,⁶ hence we format the DateTime object as a string including only hour, minute and second information (line 17). Even though we only use hours and minutes for the x-axis tick labels later, the time data is resolved down to the second, hence we retain the seconds information in the @times array.

One could write a more compact version of the time zone correction code like this:

my @times = map {
    $date_parser->parse_datetime($_)
        ->set_time_zone('Europe/Berlin')
        ->strftime("%H:%M:%S");
} @timestamps;

yet, in this case, I find giving each step a name (via a variable) helps the code explain itself. YMMV.

The next chunk of code (lines 22-23) isn’t related to the time zone fix. It generalises working out the current date from the activity data. This way I can use a FIT file from a different activity without having to update the $date variable by hand. The process is simple: all elements of the @timestamps array have the same date, so we choose to parse only the first one (line 22)⁷. This gives us a DateTime object which we convert into a formatted date string (via the strftime() method) composed of the year, month and day (line 23). We don’t need to fix the time zone because UTC is sufficient in this case to extract the date information. Of course, if you’re in a time zone close to the international date line you might need to set the time zone to get the correct date.

The last thing to change is the timefmt option to the Chart::Gnuplot::DataSet object on line 41. Because we now only have hour, minute and second information, we need to update the time format string to reflect this.

Now we’re ready to run the script again! Doing so

$ perl geo-fit-plot-data.pl

creates this graph

where we see that the time information is correct. Yay! 🎉

How long can this go on?

Now that I look at the graph again, I realise something: it doesn’t matter when the data was taken (at least, not for this use case). What matters more is the elapsed time from the start of the activity until the end. It looks like we need to munge the time data again. The job now is to convert the timestamp information into seconds elapsed since the ride began. Since we’ve parsed the timestamp data into DateTime objects (in line 15 above), we can convert that value into the number of seconds since the epoch (via the epoch() method). As soon as we know the epoch value for each element in the @timestamps array, we can subtract the first element’s epoch value from each element in the array. This will give us an array containing elapsed seconds since the beginning of the activity. Elapsed seconds are a bit too fine-grained for an activity extending over an hour, so we’ll also convert seconds to minutes.

Making these changes to the plot_activity_data() code, we get:

 1sub plot_activity_data {
 2    my @activity_data = @_;
 3
 4    # extract data to plot from full activity data
 5    my @heart_rates = num_parts('heart_rate', @activity_data);
 6    my @timestamps = map { $_->{'timestamp'} } @activity_data;
 7
 8    # parse timestamp data
 9    my $date_parser = DateTime::Format::Strptime->new(
10        pattern => "%Y-%m-%dT%H:%M:%SZ",
11        time_zone => 'UTC',
12    );
13
14    # get the epoch time for the first point in the time data
15    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;
16
17    # convert timestamp data to elapsed minutes from start of activity
18    my @times = map {
19        my $dt = $date_parser->parse_datetime($_);
20        my $epoch_time = $dt->epoch;
21        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
22        $elapsed_time;
23    } @timestamps;
24
25    # determine date from timestamp data
26    my $dt = $date_parser->parse_datetime($timestamps[0]);
27    my $date = $dt->strftime("%Y-%m-%d");
28
29    # plot data
30    my $chart = Chart::Gnuplot->new(
31        output => "watopia-figure-8-heart-rate.png",
32        title  => "Figure 8 in Watopia on $date: heart rate over time",
33        xlabel => "Elapsed time (min)",
34        ylabel => "Heart rate (bpm)",
35        terminal => "png size 1024, 768",
36    );
37
38    my $data_set = Chart::Gnuplot::DataSet->new(
39        xdata => \@times,
40        ydata => \@heart_rates,
41        style => "lines",
42    );
43
44    $chart->plot2d($data_set);
45}

The main changes occur in lines 14-23. We parse the date/time information from the first timestamp (line 15), chaining the epoch() method call to find the number of seconds since the epoch. We store this result in a variable for later use; it holds the epoch time at the beginning of the data series. After parsing the timestamps into DateTime objects (line 19), we find the epoch time for each time point (line 20). Line 21 calculates the elapsed time from the time stored in $first_epoch_time and converts seconds to minutes by dividing by 60. The map returns this value (line 22) and hence @times now contains a series of elapsed time values in minutes.

It’s important to note here that we’re no longer plotting a date/time value on the x-axis; the elapsed time is a purely numerical value. Thus, we update the x-axis label string (line 33) to highlight this fact and remove the timeaxis and xtics/labelfmt options from the Chart::Gnuplot constructor. The timefmt option to the Chart::Gnuplot::DataSet constructor is also no longer necessary and it too has been removed.

The script is now ready to go!

Running it

$ perl geo-fit-plot-data.pl

gives

That’s better!

Reaching for the sky

Our statistics output from earlier told us that the maximum heart rate was 165 bpm with an average of 142 bpm. Looking at the graph, an average of 142 bpm seems about right. It also looks like the maximum pulse value occurred at an elapsed time of just short of 50 minutes. We can check that guess more closely later.

What’s intriguing me now is what caused this pattern in the heart rate data. What could have caused the values to go up and down like that? Is there a correlation with other data fields? We know from earlier that there’s an altitude field, so we can try plotting that along with the heart rate data and see how (or if) they’re related.

Careful readers might have noticed something: how can you have a variation in altitude when you’re sitting on an indoor trainer? Well, Zwift simulates going up and downhill by changing the resistance in the smart trainer. The resistance is then correlated to a gradient and, given time and speed data, one can work out a virtual altitude gain or loss. Thus, for the data set we’re analysing here, altitude is a sensible parameter to consider. Even if you had no vertical motion whatsoever!

As I mentioned earlier, one of the things I like about Gnuplot is that one can plot two data sets with different y-axes on the same plot. Plotting heart rate and altitude on the same graph is one such use case.

To plot an extra data set on our graph, we need to set up another Chart::Gnuplot::DataSet object, this time for the altitude data. Before we can do that, we’ll have to extract the altitude data from the full activity data set. Gnuplot also needs to know which data to plot on the primary and secondary y-axes (i.e. on the left- and right-hand sides of the graph). And we must remember to label our axes properly. That’s a fair bit of work, so I’ve done the hard yakka for ya. 😉

Here’s the updated plot_activity_data() code:

 1sub plot_activity_data {
 2    my @activity_data = @_;
 3
 4    # extract data to plot from full activity data
 5    my @heart_rates = num_parts('heart_rate', @activity_data);
 6    my @timestamps = map { $_->{'timestamp'} } @activity_data;
 7    my @altitudes = num_parts('altitude', @activity_data);
 8
 9    # parse timestamp data
10    my $date_parser = DateTime::Format::Strptime->new(
11        pattern => "%Y-%m-%dT%H:%M:%SZ",
12        time_zone => 'UTC',
13    );
14
15    # get the epoch time for the first point in the time data
16    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;
17
18    # convert timestamp data to elapsed minutes from start of activity
19    my @times = map {
20        my $dt = $date_parser->parse_datetime($_);
21        my $epoch_time = $dt->epoch;
22        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
23        $elapsed_time;
24    } @timestamps;
25
26    # determine date from timestamp data
27    my $dt = $date_parser->parse_datetime($timestamps[0]);
28    my $date = $dt->strftime("%Y-%m-%d");
29
30    # plot data
31    my $chart = Chart::Gnuplot->new(
32        output => "watopia-figure-8-heart-rate-and-altitude.png",
33        title  => "Figure 8 in Watopia on $date: heart rate and altitude over time",
34        xlabel => "Elapsed time (min)",
35        ylabel => "Heart rate (bpm)",
36        terminal => "png size 1024, 768",
37        xtics => {
38            incr => 5,
39        },
40        y2label => 'Altitude (m)',
41        y2range => [-10, 70],
42        y2tics => {
43            incr => 10,
44        },
45    );
46
47    my $heart_rate_ds = Chart::Gnuplot::DataSet->new(
48        xdata => \@times,
49        ydata => \@heart_rates,
50        style => "lines",
51    );
52
53    my $altitude_ds = Chart::Gnuplot::DataSet->new(
54        xdata => \@times,
55        ydata => \@altitudes,
56        style => "boxes",
57        axes => "x1y2",
58    );
59
60    $chart->plot2d($altitude_ds, $heart_rate_ds);
61}

Line 7 extracts the altitude data from the full activity data. This code also strips the unit information from the altitude data so that we only have the numerical part, which is what Gnuplot needs. We store the altitude data in the @altitudes array. This we use later to create a Chart::Gnuplot::DataSet object on lines 53-58. An important line to note here is the axes setting on line 57; it tells Gnuplot to use the secondary y-axis on the right-hand side for this data set. I’ve chosen to use the boxes style for the altitude data (line 56) so that the output looks a bit like the hills and valleys that it represents.

To make the time data a bit easier to read and analyse, I’ve set the increment for the ticks on the x-axis to 5 (lines 37-39). This way it’ll be easier to refer to specific changes in altitude and heart rate data.

The settings for the secondary y-axis use the same names as for the primary y-axis, with the exception that the string y2 replaces y. For instance, to set the axis label for the secondary y-axis, we specify the y2label value, as in line 40 above.

I’ve set the range on the secondary y-axis explicitly (line 41) because the output looks better than what the automatic range was able to make in this case. Similarly, I’ve set the increment on the secondary y-axis ticks (lines 42-44) because the automatic output wasn’t as good as what I wanted.

I’ve also renamed the variable for the heart rate data set on line 47 to be more descriptive; the name $data_set was much too generic.

We specify the altitude data set first in the call to plot2d() (line 60) because we want the heart rate data plotted “on top” of the altitude data. Had we used $heart_rate_ds first in this call, the altitude data would have obscured part of the heart rate data.

Running our script in the now familiar way

$ perl geo-fit-plot-data.pl

gives this plot

Cool! Now it’s a bit clearer why the heart rate evolved the way it did.

At the beginning of the graph (in the first ~10 minutes) it looks like I was getting warmed up and my pulse was finding a kind of base level (~130 bpm). Then things started going uphill at about the 10-minute mark and my pulse also kept going upwards. This makes sense because I was working harder. Between about 13 minutes and 19 minutes came the first hill climb on the route and here I was riding even harder. The effort is reflected in the heart rate data which rose to around 160 bpm at the top of the hill. That explains why the heart rate went up from the beginning to roughly the 18-minute mark.

Looking back over the Zwift data for that particular ride, it seems that I took the KOM⁸ for that climb at that time, so no wonder my pulse was high!⁹ Note that this wasn’t a special record or anything like that; it was a short-term live result¹⁰ and someone else took the jersey with a faster time not long after I’d done my best time up that climb.

It was all downhill shortly after the hill climb, which also explains why the heart rate went down straight afterwards. We also see similar behaviour on the second hill climb (from about 37 minutes to 42 minutes). Although my pulse rose throughout the hill climb, it didn’t rise as high this time. This indicates that I was getting tired and wasn’t able to put as much effort in.

Just in case you’re wondering how the altitude can go negative,¹¹ part of the route goes through “underwater tunnels”. This highlights the flexibility of the virtual worlds within Zwift: the designers have enormous room to let their imaginations run wild. There are all kinds of fun things to discover along the various routes and many that don’t exist in the Real World™. Along with the underwater tunnels (where it’s like riding through a giant aquarium, with sunken shipwrecks, fish, and whales), there is a wild west style town complete with a steam train from that era chugging past. There are also Mayan ruins with llamas (or maybe alpacas?) wandering around and even a section with dinosaurs grazing at the side of the road.

Here’s what it looks like riding through an underwater tunnel:

I think that’s pretty cool.

At the end of the ride (at ~53 minutes) my pulse dropped sharply. Since this was the warm-down phase of the ride, this also makes sense.

Power to the people!

There are two peaks in the heart rate data that don’t correlate with altitude (one at ~25 minutes and another at ~48 minutes). The altitude change at these locations would suggest that things are fairly flat. What’s going on there?

One other parameter that we could consider for correlations is power output. Going uphill requires more power than riding on the flat, so we’d expect to see higher power values (and therefore higher heart rates) when climbing. If flat roads require less power, what’s causing the peaks in the pulse? Maybe there’s another puzzle hiding in the data.

Let’s combine the heart rate data with power output and see what other relationships we can discover. To do this we need to extract power output data instead of altitude data. Then we need to change the secondary y-axis data set and configuration to produce a nice plot of power output. Making these changes gives this code:

 1sub plot_activity_data {
 2    my @activity_data = @_;
 3
 4    # extract data to plot from full activity data
 5    my @heart_rates = num_parts('heart_rate', @activity_data);
 6    my @timestamps = map { $_->{'timestamp'} } @activity_data;
 7    my @powers = num_parts('power', @activity_data);
 8
 9    # parse timestamp data
10    my $date_parser = DateTime::Format::Strptime->new(
11        pattern => "%Y-%m-%dT%H:%M:%SZ",
12        time_zone => 'UTC',
13    );
14
15    # get the epoch time for the first point in the time data
16    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;
17
18    # convert timestamp data to elapsed minutes from start of activity
19    my @times = map {
20        my $dt = $date_parser->parse_datetime($_);
21        my $epoch_time = $dt->epoch;
22        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
23        $elapsed_time;
24    } @timestamps;
25
26    # determine date from timestamp data
27    my $dt = $date_parser->parse_datetime($timestamps[0]);
28    my $date = $dt->strftime("%Y-%m-%d");
29
30    # plot data
31    my $chart = Chart::Gnuplot->new(
32        output => "watopia-figure-8-heart-rate-and-power.png",
33        title  => "Figure 8 in Watopia on $date: heart rate and power over time",
34        xlabel => "Elapsed time (min)",
35        ylabel => "Heart rate (bpm)",
36        terminal => "png size 1024, 768",
37        xtics => {
38            incr => 5,
39        },
40        ytics => {
41            mirror => "off",
42        },
43        y2label => 'Power (W)',
44        y2range => [0, 1100],
45        y2tics => {
46            incr => 100,
47        },
48    );
49
50    my $heart_rate_ds = Chart::Gnuplot::DataSet->new(
51        xdata => \@times,
52        ydata => \@heart_rates,
53        style => "lines",
54    );
55
56    my $power_ds = Chart::Gnuplot::DataSet->new(
57        xdata => \@times,
58        ydata => \@powers,
59        style => "lines",
60        axes => "x1y2",
61    );
62
63    $chart->plot2d($power_ds, $heart_rate_ds);
64}

On line 7, I swapped out the altitude data extraction code with power output. Then, I updated the output filename (line 32) and plot title (line 33) to highlight that we’re now plotting heart rate and power data.

The mirror option to the ytics setting (lines 40-42) isn’t an obvious change. Its purpose is to stop the ticks from the primary y-axis from being mirrored to the secondary y-axis (on the right-hand side). We want to stop these mirrored ticks from appearing because they’ll clash with the secondary y-axis tick marks. The reason we didn’t need this before is that all the y-axis ticks happened to line up and the issue wasn’t obvious until now.

I’ve updated the secondary axis label setting to mention power (line 43). Also, I’ve set the range to match the data we’re plotting (line 44) and to space out the data nicely via the incr option to the y2tics setting (lines 45-47). It seemed more appropriate to use lines to plot power output as opposed to the bars we used for the altitude data, hence the change to the style option on line 59.

As we did when plotting altitude, we pass the power data set ($power_ds) to the plot2d() call before $heart_rate_ds (line 63).

Running the script again

$ perl geo-fit-plot-data.pl

produces this plot:

This plot shows the correlation between heart rate and power output that we expected for the first hill climb. The power output increases steadily from the 3-minute mark up to about the 18-minute mark. After that, it dropped suddenly once I’d reached the top of the climb. This makes sense: I’d just done a personal best up that climb and needed a bit of respite!

However, now we can see clearly what caused the spikes in heart rate at 25 minutes and 48 minutes: there are two large spikes in power output. The first spike maxes out at 1023 W;¹² what value the other peak has, it’s hard to tell. We’ll try to work out what that value is later. These spikes in power result from sprints. In Zwift, not only can one try to go up hills as fast as possible, but flatter sections have sprints where one also tries to go as fast as possible, albeit for shorter distances (say 200m or 500m).

Great! We’ve worked out another puzzle in the data!

A quick comparison

Zwift produces what they call timelines of a given ride, which is much the same as what we’ve been plotting here. For instance, for the FIT file we’ve been looking at, this is the timeline graph:

Zwift plots several datasets on this graph that have very different value ranges. The plot above shows power output, cadence, heart rate, and altitude data all on one graph! A lot is going on here and because of the different data values and ranges, Zwift doesn’t display values on the y-axes. Their solution is to show all four values at a given time point when the user hovers their mouse over the graph. This solution only works within a web browser and needs lots of JavaScript to work, hence this is something I like to avoid. That (and familiarity) is largely the reason why I prefer PNG output for my graphs.

If you take a close look at the timeline graph, you’ll notice that the maximum power is given as 937 W and not 1023 W, which we worked out from the FIT file data. I don’t know what’s going on here, as the same graph in the Zwift Companion App shows the 1023 W that we got. The graph above is a screenshot from the web application in a browser on my laptop and, at least theoretically, it’s supposed to display the same data. I’ve noticed a few inconsistencies between the web browser view and that from the Zwift Companion App, so maybe this discrepancy is one bug that still needs shaking out.

A dialogue with data

Y’know what’d also be cool beyond plotting this data? Playing around with it interactively.

That’s also possible with Perl, but it’s another story.

I’ve been using Gnuplot since the late 90’s. Back then, it was the only freely available plotting software which handled time data well. ↩︎
By default, Gnuplot will generate Postscript output. ↩︎
One can interpret the word “terminal” as a kind of “screen” or “canvas” that the plotting library draws its output on. ↩︎
I’ve later found out that they haven’t heard anything, so that’s good! ↩︎
I live in Germany, so this is the relevant time zone for me. ↩︎
All dates are the same and displaying them would be redundant, hence we omit the date information. ↩︎
All elements in the array have the same date, so using the first one does the job. ↩︎
KOM stands for “king of the mountains”. ↩︎
Yes, I am stoked that I managed to take that jersey! Even if it was only for a short time. ↩︎
A live result that makes it onto a leaderboard is valid only for one hour. ↩︎
Around the 5-minute mark and again shortly before the 35-minute mark. ↩︎
One thing that this value implies is that I could power a small bar heater for one second. But not for very much longer! ↩︎

Articles on Perl.com - programming news, code and culture

Making an Asynchronous Clocking Drum Machine App in Perl

Clocks

Grooves

The Module

Credits

To-do Challenges

Resources

Musical Rhythms with Math in Perl

Sidebar

So what have we learned today?

References:

Fastmail Donates USD 10,000 to The Perl and Raku Foundation

Cast-Iron Community: Your Chance to Sponsor TPRC 2026

About the Conference

Why Sponsor?

Sponsorship Tiers

Platinum Sponsor ($6,000)

Gold Sponsor ($4,000)

Silver Sponsor ($2,000)

Bronze Sponsor ($1,000)

All Sponsors Receive

Named Sponsorship Opportunities

About The Perl and Raku Foundation

Contact Information

Podlite comes to Perl: a lightweight block-based markup language for everyday use

The Block Structure of Podlite

Abbreviated blocks (=BLOCK)

Paragraph blocks (=for BLOCK)

Delimited blocks (=begin BLOCK … =end BLOCK)

Example: Comparing POD and Podlite

Inside the Podlite Specification 1.0

Using Podlite in Perl via the source filter

Roadmap: what’s next for Podlite

Try Podlite and share feedback

Announcing the Perl Toolchain Summit 2026

Announcing the Perl Toolchain Summit 2026!

DuckDuckGo Donates $25,000 to The Perl and Raku Foundation v2025

Geizhals Preisvergleich Donates USD 10,000 to The Perl and Raku Foundation

Announcing Dancer2 2.0.0

Your Favorite Perl Web Framework, Now Even Better

Release Highlights

Thank You to Our Contributors

What’s Next?

Keep Dancing!

My Guilty Perl Obsession

The Guilt

The Pleasure

Holophrastic Enterprises Inc.

SUSE Donates USD 11,500 to The Perl and Raku Foundation

Perl is a Fundamental Component of the SUSE Ecosystem

The Perl-Powered OpenQA and Open Build Service

Sustaining the Digital Commons

A Partnership Model for Open Source Sustainability

MetaCPAN's Traffic Crisis: An Eventual Success Story

The Problem Emerges

Initial Investigation Phase

The Datadog Breakthrough

Escalating Defences

Next-Generation WAF Implementation

The Dynamic Challenge Solution

Current State: Victory Through Data

Community Heroes

Analysing FIT data with Perl: interactive data analysis

Interactive data analysis

Getting set up

Cleaning up to continue

Making a mini-module

Poking and prodding

Looking forward

Open question

Conclusion

Proxmox Donates €10,000 to The Perl and Raku Foundation

A Partner in Open Source

Sustaining a Foundation of Modern Computing

The Critical Nature of Core Maintenance

A Shared Vision for Open Source Sustainability

Looking Forward

Analysing FIT data with Perl: producing PNG plots

Start plotting with Gnu

Abbreviated blocks (`=BLOCK`)

Paragraph blocks (`=for BLOCK`)

Delimited blocks (`=begin BLOCK` … `=end BLOCK`)