## Introduction

I recently migrated to the Exetel One Plan. That 500Mbps was hard to turn down coming in at both a faster speed and lower cost than my current plan. There was one central issue though: CG-NAT.

Exetel's old plans allowed customers to opt-out of CG-NAT, which was key in my remote access setup. I forwarded port 51820 to my home-server on my router and very occasionally changed the A record of my DNS provider. This is no longer possible and so I need a new solution for remote access.

I originally chose a WireGuard based solution for it's smallest attack surface, ease of setup, and excellent performance; I standby that decision. I need a new method for gaining access to my home-server, and fast; I just got a new phone and want to hit the ground running with device data backups via Syncthing. So what do we do?

### Approaches

There's a few options I've been playing with:
- **1. VPS Bastion (Forwarding):** A WireGuard-enabled VPS that forwards traffic to my home network.
- **2. VPS Bastion (Hole Punching Broker):** A WireGuard peer that facilitates NAT traversal and endpoint discovery, with peer-to-peer connection to the home server.
- **3. IPv6 Direct Access:** WireGuard over native IPv6 to my home server (requires IPv6-capable router and network).

Option 1 is the simplest and quickest to implement — just a new VPS and some config changes.

Option 2 is more complex, but potentially the highest performance. I imagine a setup where a Caddy server (on the VPS) sits behind an [[Authelia]] instance. After authentication, Caddy issues redirects to my home server (once a UDP hole-punched connection is established) via WireGuard. This would allow low-latency, direct access with minimal resource usage. I found a good reference for the hole-punching setup in [1].

Option 3 is possible, but not without additional hardware and networking reconfiguration, so it’s out for now.

For the time being, I think we go with Option 1, and migrate to option two at a later date as that gets us up and running sooner.

But wait. Why not just use something like Tailscale? Fair question, I absolutely could. Tailscale would be simple and fast to setup. But to me, that approach doesn't seem ultimately scalable or sovereign. I want a solution that is 100% independent, self-hosted, and reproducible — one I control end-to-end. My goal is to eventually have fully automated provisioning using NixOS configs, where everything from tunnel setup to access control is declarative and transparent.

So while Tailscale is technically viable, it doesn't align with the kind of system I want to build. Option 1 it is.

## Bootstrapping a NixOS VPS
To kick off the remote access process, we need to setup a VPS running NixOS. I'm already running a few services on DigitalOcean so the plan is to setup a service here and eventually consolidate all my services to the single droplet to save costs.

DigitalOcean doesn't offer a NixOS image. In fact, very few VPS providers do. There is however, a community developed solution to this problem: [`nixos-anywhere`] [nixos-anywhere]. This a tool that allows for unattended setup and installation of NixOS via SSH on any Linux based machine. It is nothing short of excellent.

I followed this [nix.dev guide][nix-remote-prov] for setting up my machine. I opted for a droplet with 2GB of RAM and 1vCPU, this was the cheapest I could get; the `nixos-anywhere` docs require a machine with at least 2GB of RAM. I additionally enabled IPv6.

I worked through the steps in the guide and managed to provision my machine. I was however unanble to SSH into it. It turned out that DigitalOcean droplets require their networking configuration be manually setup.

This proved a little troubling for me, but I here's how I got around it. While searching for options to provision NixOS on a VPS, I found [this article][nixos-docean] which linked a repo called [nixos-infect][nixos-infectj]. It was similar in concept to NixOS anywhere, but far less reliable. I tried installing it a few times, but each time I would end up with a broken NixOS installation; I couldn't rebuild switch and update my configs. But interestingly networking worked.  The `nixos-infect` script spat out a little `networking.nix`  configuration file. This little fact was stored away in my brain while I had moved on to working with `nixos-anywhere`.

So when networking didn't work I had a little light-bulb moment. Why not use the networking config generation script to generate the `networking.nix` config and import that into our `nixos-anywhere` installation? Magic .

I pulled the networking config generation out of the project and into bash script:
```sh
makeNetworkingConf() {
  # XXX It'd be better if we used procfs for all this...
  local IFS=$'\n'
  eth0_name=$(ip address show | grep '^2:' | awk -F': ' '{print $2}')
  eth0_ip4s=$(ip address show dev "$eth0_name" | grep 'inet ' | sed -r 's|.*inet ([0-9.]+)/([0-9]+).*|{ address="\1"; prefixLength=\2; }|')
  eth0_ip6s=$(ip address show dev "$eth0_name" | grep 'inet6 ' | sed -r 's|.*inet6 ([0-9a-f:]+)/([0-9]+).*|{ address="\1"; prefixLength=\2; }|' || '')
  gateway=$(ip route show dev "$eth0_name" | grep default | sed -r 's|default via ([0-9.]+).*|\1|')
  gateway6=$(ip -6 route show dev "$eth0_name" | grep default | sed -r 's|default via ([0-9a-f:]+).*|\1|' || true)
  ether0=$(ip address show dev "$eth0_name" | grep link/ether | sed -r 's|.*link/ether ([0-9a-f:]+) .*|\1|')

  eth1_name=$(ip address show | grep '^3:' | awk -F': ' '{print $2}')||true
  if [ -n "$eth1_name" ];then
    eth1_ip4s=$(ip address show dev "$eth1_name" | grep 'inet ' | sed -r 's|.*inet ([0-9.]+)/([0-9]+).*|{ address="\1"; prefixLength=\2; }|')
    eth1_ip6s=$(ip address show dev "$eth1_name" | grep 'inet6 ' | sed -r 's|.*inet6 ([0-9a-f:]+)/([0-9]+).*|{ address="\1"; prefixLength=\2; }|' || '')
    ether1=$(ip address show dev "$eth1_name" | grep link/ether | sed -r 's|.*link/ether ([0-9a-f:]+) .*|\1|')
    interfaces1=$(cat << EOF
      $eth1_name = {
        ipv4.addresses = [$(for a in "${eth1_ip4s[@]}"; do echo -n "
          $a"; done)
        ];
        ipv6.addresses = [$(for a in "${eth1_ip6s[@]}"; do echo -n "
          $a"; done)
        ];
        };
EOF
)
    extraRules1="ATTR{address}==\"${ether1}\", NAME=\"${eth1_name}\""
  else
    interfaces1=""
    extraRules1=""
  fi

  readarray nameservers < <(grep ^nameserver /etc/resolv.conf | sed -r \
    -e 's/^nameserver[[:space:]]+([0-9.a-fA-F:]+).*/"\1"/' \
    -e 's/127[0-9.]+/8.8.8.8/' \
    -e 's/::1/8.8.8.8/' )

  if [[ "$eth0_name" = eth* ]]; then
    predictable_inames="usePredictableInterfaceNames = lib.mkForce false;"
  else
    predictable_inames="usePredictableInterfaceNames = lib.mkForce true;"
  fi
  cat > networking.nix << EOF
{ lib, ... }: {
  # This file was populated at runtime with the networking
  # details gathered from the active system.
  networking = {
    nameservers = [ ${nameservers[@]} ];
    defaultGateway = "${gateway}";
    defaultGateway6 = {
      address = "${gateway6}";
      interface = "${eth0_name}";
    };
    dhcpcd.enable = false;
    $predictable_inames
    interfaces = {
      $eth0_name = {
        ipv4.addresses = [$(for a in "${eth0_ip4s[@]}"; do echo -n "
          $a"; done)
        ];
        ipv6.addresses = [$(for a in "${eth0_ip6s[@]}"; do echo -n "
          $a"; done)
        ];
        ipv4.routes = [ { address = "${gateway}"; prefixLength = 32; } ];
        ipv6.routes = [ { address = "${gateway6}"; prefixLength = 128; } ];
      };
      $interfaces1
    };
  };
  services.udev.extraRules = ''
    ATTR{address}=="${ether0}", NAME="${eth0_name}"
    $extraRules1
  '';
}
EOF
}

makeNetworkingConf
```

I created a new Ubuntu droplet, copied the script across and ran it. I imported the `networking.nix` file into my `nixos-anywhere` config:
```diff
{ modulesPath, ... }:

let
  diskDevice = "/dev/vda";
  sources = import ./npins;
in
{
  imports = [
    (modulesPath + "/profiles/qemu-guest.nix")
    (sources.disko + "/module.nix")
    ./single-disk-layout.nix
+   ./networking.nix
  ];

  disko.devices.disk.main.device = diskDevice;

  boot.loader.grub = {
    devices = [ diskDevice ];
    efiSupport = true;
    efiInstallAsRemovable = true;
  };

  services.openssh.enable = true;

  users.users.root.openssh.authorizedKeys.keys = [
	  ...
  ];

  system.stateVersion = "24.11";
}
```

I re-ran the installation process:
```o
toplevel=$(nixos-rebuild build --no-flake) \
diskoScript=$(nix-build -E "((import <nixpkgs> {}).nixos [ ./configuration.nix ]).diskoScript") \
nixos-anywhere --store-paths "$diskoScript" "$toplevel" root@my-droplet-ip
```

And that was it. SSH access worked. We had successfully bootstrapped NixOS onto our VPS. Onto the WireGuard setup.

## WireGuard Bastion Setup

The WireGuard bastion setup proved a little challenging due to my lack of deep understanding of the application; I had only used it for point-to-site connections in the past. I found this [how-to][wireguard-sharing-bastion] and it helped me get everything up and running. It didn't really do an excellent job of explaining the *why*, so my plan is to do that here.

First, the goal: two machines (remotes) communicating via another machine (bastion). Each machine runs an instance of Wireguard. There are two styles of WireGuard configurations in this setup, the `remote` configs and the `bastion` configs. The `remote` configs are nearly identical having only different keys.

The `bastion` config is shown below (written in `nix`):
```nix
{config, ...}: {
  networking = {
    wireguard.enable = true;
    wireguard.interfaces = {
      wg0 = {
        ips = ["10.0.0.1/24"];
        listenPort = 51820;
        privateKeyFile = "/etc/wireguard/private.key";

        peers = [
          {
            # Pixel 8a
            publicKey = "...";
            allowedIPs = ["10.0.0.2/32"];
          }
          {
            # Myshkin
            publicKey = "...";
            allowedIPs = ["10.0.0.3/32"];
          }
        ];
      };
    };

    firewall.allowedUDPPorts = [51820];
  };
}
```

Note how this is setup, we have a list of peers, each assigned a fully-qualified IP on the subnet: `10.0.0.x/32`. Note also the IPs assigned to our interface: `10.0.0.1/24`. In practice this means that our bastion can received traffic from any of the defined clients.

Now, look at a `remote` style config:
```nix
{config, ...}: {
  networking = {
    wireguard.enable = true;
    wireguard.interfaces = {
      wg0 = {
        # Determines the IP address and subnet of the server's end of the tunnel interface.
        ips = ["10.0.0.3/24"];
        listenPort = 51820;
        privateKeyFile = config.sops.secrets."wireguard/myshkin/private-key".path;

        peers = [
          {
            # Shatov Jump Host
            endpoint = "ip-and-port-of-our-bastion";
            publicKey = "...":
            allowedIPs = ["10.0.0.0/24"];
            persistentKeepalive = 25;
          }
        ];
      };
    };

    firewall.allowedUDPPorts = [51820];
  };
}

```

Note in particular the `allowedIPs` field. It is set to `10.0.0.0/24`. This means that it will allow any traffic on the `10.0.0.0/24` subnet, including our remote devices, which this peer has no knowledge of.  The remotes are not remote-aware, but they will accept traffic if it arrives from the bastion. Note also that I've setup a `persistentKeepAlive` so that my home-server is always connected to the bastion.

All that's left now is to explicitly enable IPv4 forwarding on the VPS. This can be done by adding the following to your NixOS config.

```nix
boot.kernel.sysctl = {
  "net.ipv4.conf.all.forwarding" = true;
}
```

And that's it. With a config like this, you have a simple and secure method for accessing your home server behind a NAT from any remote device. Now I just need to setup Syncthing.


[1]:https://www.jordanwhited.com/posts/wireguard-endpoint-discovery-nat-traversal/
[nixos-anywhere]:https://github.com/nix-community/nixos-anywhere
[nix-remote-prov]:[https://nix.dev/tutorials/nixos/provisioning-remote-machines.html
[nixos-docean]:https://usagi.io/articles/2024-04-14-deploying-nixos-to-digital-ocean/
[nixos-infect]:https://github.com/elitak/nixos-infect/blob/master/nixos-infect
[nixos-home-router]:https://francis.begyn.be/blog/nixos-home-router

VPS Remote Access Bastion using NixOS and WireGuard

Making a qBittorrent Module for NixOS

Tinix I -- Porting ESP-Hosted for the Luckfox Pico

I've had a Kindle Touch 5 (KT5) since they were released, around 2012. I've
used it on and off over the years, and it's still rock solid. I can't really
fault the device, it's the only piece of hardware that I've owned for 10+ years
that still works without issue.

Despite not having any issues, I've recently become interested in Jailbreaking
my device. The main motivation for this is gaining root access to the device
and installing [KOReader][1] which appears to be a much improved eink reader 
app. Jailbreaking my device should allow for better integration with my [home
server setup][2], without purchasing a new/different device.

## Finding the right jailbreak 
Most of the kindle jailbreaking info is on the [MobileRead forums][3]. The main
issue is figuring out which jailbreak to use. This depends both on the device
you are using and the installed firmware.

See [here][4] for identifying the device. Once you've identified the device,
I suggest checking out this [snapshots][5] page. It basically aggregates all
the kindle jailbreaks into a single location. Using both your device firmware
and device version, you should be able to identify the most suitable jailbreak.

Just search for that and you should be able to find an appropriate MobileRead
post with instructions.

## Jailbreaking
For my device and firmware I found the following to be most suitable:
- K5 JailBreak (5.0.x - 5.4.4.2)

This is available [here][6]. This is a neat little jailbreak, allowing me to root
the device without factory resetting.

I'll reproduce the instructions below for posterity:

---

Download the K5 JailBreak package from the Snapshots thread,
and unpack it. In here, you'll find a few files.
Unzip the contents of the kindle-5.4-jailbreak.zip archive
to the root directory of your Kindle.

Now, eject & unplug your Kindle, and go to
[HOME] -> [MENU] > Settings -> [MENU] > Update Your Kindle.
The updater won't actually ever run, this is normal.
After a few seconds, the words **** JAILBREAK **** will appear
at the bottom of your screen to confirm that the device is jailbroken.
You won't see this message again, it's simply a confirmation message.

And that's it, your Kindle is now ready to install custom hacks!

Since v1.11.N: If you need to use something MKK-powered immediately,
be aware that MKK will only be active on the next boot.
Installing another package through the Update mechanism will do the trick,
or you can manually trigger a device restart: > ([HOME] -> [MENU] >
Settings -> [MENU] > Restart).

---

I should note that the jailbreak worked. Now to install some new software.

## KUAL
Most of the the third-party kindle software relies on [KUAL][7]. This is an
app launcher for kindle, and is usually a requirement for any apps. Installing
this is straightforward. I've reproduced the instructions below.

---
... you may need to install MKK: if your device runs on FW 5.x, and you're
using the K5 JB >= 1.11.N (or the associated "hotfix"), you don't need to do
anything, MKK is bundled with your JB. If you're using an older device, you
will need to install MKK yourself 

Finally, you need to install KUAL itself, by putting the appropriate azw2
file somewhere in the documents/ directory tree of your device.
First, download the latest KUAL archive from the Snapshots thread and unpack it.
Then, if your device runs on FW 5.x, use the KUAL-KDK-2.0.azw2 file. If your
device is older, use the KUAL-KDK-1.0.azw2 file.

And that's it for the basics!

---
For my jailbreak, MKK came installed, meaning I can just copy the azw2 file to
the documents directory.

I did this, then tried to launch KUAL immediately and failed. Turns out I
needed to reboot my device to properly install the jailbreak. After rebooting,
KUAL launched as expected!

## KUAL Extensions!
KUAL isn't really much on it's own. Instead, it aggregrates extensions. Let's
start by installing the recommended extensions!

First I installed Helper and blocked OTA updates. Silent OTA updates might
break the device...

Next I installed MRPI.

## KOReader
To install KOReader, you can follow the instructions [here][8]. In short,
download the right release and unzip to the root directoty. I used the `kindle`
release.

I was initially a little bit underwhelmed by KOReader. I tried reading my
`.mobi` files and they were atrociously slow and poorly styled. Note that these
were automatically converted from the `.epub` source by Calibre. Given that
`.epub` support was part of my motivation for switching, I downloaded the same
book [Rhythm of War - Grant Sanderson] in epub format onto my Kindle. Wow.
The speed of epubs was outstanding and they maintained the correct formatting.

I customised the look and feel--line spacing, margins, alt status--downloaded
dictionaries, and looked through some of the menus. I'm really impressed. There
are some great features, like the reading statistics and plugins, that really
sell this as a major upgraded for an otherwise very old kindle.

## Conclusion
If you're on the fence definitely give jailbreaking a go. I highly recommend
KOReader. There are some quirks, like menu overlay issues on waking, but the
experience is much better overall than the stock kindle reader. You get
granular control over text rendering, rich format support, better reading
statistics and much more.

[1]:https://koreader.rocks/
[2]:https://www.samkwort.com/home-server-v3
[3]:https://www.mobileread.com/forums/forumdisplay.php?f=150
[4]:https://wiki.mobileread.com/wiki/Kindle_Serial_Numbers
[5]:https://www.mobileread.com/forums/showthread.php?t=225030
[6]:https://www.mobileread.com/forums/showthread.php?t=186645
[7]:https://www.mobileread.com/forums/showthread.php?t=203326
[8]:https://github.com/koreader/koreader/wiki/Installation-on-Kindle-devices


Jailbreaking a Kindle Touch 5

You may or may not have noticed that the website has changed. I built the old site,
*samkwort.com*, on a whim back in June/July 2020. I had pretty limited experience
with web development and wanted to get some familiarity with technologies that I
deemed to be quite accessible at the time. That old site was built using *Pelican*,
a static site generator implemented in Python. I remember thinking at the time that
I wanted to build a website without using any JavaScript. I was successful, though
I'm not sure why I thought that. Fast-forward three years and that old site started
to look pretty dated. Beyond that, I couldn't remember, nor did I document, how to
add new posts to the website. I figured I might as well rebuild it using modern web
technologies and take a deeper dive into the world of web development through
JavaScript.

## Goals
I wanted my new site to look better and be easier to create content for. I also
wanted to learn a bit about JavaScript. My experience had been limited to writing
a very small web scraper + JSON data server for an ESP32 project I've been working
on recently. I also wanted to learn a bit more about React; I was introduced to
React during an internship during the summer break--we were tasked with building
a custom front-end for generating reports based on Jenkins CI runs. I didn't
touch React, doing most of the backend Python-based data scraping. React seemed
pretty cool at that time. I actually sat across from a senior dev who happened
to be working on a front-end rewrite of an old Ember website. He kept saying that
any new JavaScript projects should really be written in TypeScript. I added that
to the list of things I wanted to learn.


## Implementation
I did some research into modern web frameworks and settled on *Next.js*; its
page-based routing and templating feature seemed quite intriguing, as well as
its relationship with React. It also seemed to be a bit of an industry standard.

### Layout
I started with the basic *create-next-app* template. I built the basic home-page,
creating custom *Header* and *Footer* components. Then, I created a *Layout*
component that would serve as the base for all my pages. Along the way, I picked
up a little bit of *Tailwind CSS*--styling my pages this way felt really simple.

The *Layout* element code looks like this:
```typescript
interface LayoutProps {
  children: React.ReactNode;
}

export default function Layout({ children }: LayoutProps): JSX.Element {
  return (
    <div className="mx-4 md:mx-auto md:max-w-6xl flex min-h-screen flex-col items-left p-4 md:p-24 font-mono">
      <Header/>
      <main>{children}</main>
      <Footer />
    </div>
  );
}
```
I really like the way React feels like HTML; the idea of custom components is
quite neat.

### PostList
I wanted the home-page to have a list of the most recent posts. To do this, I needed
to find a tool for parsing Markdown documents. After some research, I found the
*gray-matter* package. This parsed the Markdown into a JSON object containing data
and content fields; the data fields were generated using the YAML frontmatter of my
Markdown documents. 

I placed all my posts in a *posts* folder and wrote some code to generate a list of
Gray Matter objects. These were then sorted by the date field. I used
*getStaticProps* to ensure that the static *blogPosts* data was loaded before the
page was rendered.

Here's an excerpt of the code:

```typescript
interface HomeProps {
  postsData: Post[];
}

export async function getStaticProps(): Promise<{ props: HomeProps }> {
  const postsData = await getPostData();

  return {
    props: {
      postsData
    }
  };
}

export default function Home({ postsData }: HomeProps): JSX.Element {
  return (
    <Layout>
      <section>
        <h2 className="text-lg">Latest Posts</h2>
        <PostList postsData={postsData.slice(0, 5)} />
      </section>
    </Layout>
  );
}
```

### Dynamic routing with \[post\].tsx
*Next* offers dynamic routing, which makes it great for generating generic blog
post pages for any given Markdown document. Setting this up is fairly easy; the
[Next documentation][1] gives a pretty good example.

Once I had dynamic routes working and returning a basic hello-world type message,
I started working on parsing the Markdown blog posts into an abstracted
*BlogPost* component.

### Markdown formatting
The essence of the *BlogPost* component is Markdown to HTML conversion. I found
the perfect React component for this, [React Markdown][2]. This component takes
Markdown formatted text as its child and converts it to HTML.

I also wanted the option to display code and math in my blog posts. To set this
up, I followed the examples on the *React Markdown* GitHub page. The code
highlighting used the Syntax Highlighter package, while the math formatting
used remarkMath and rehypeKatex.

The final *BlogPost* component looks like this:

```typescript
<ReactMarkdown
  className="markdown"
  remarkPlugins={[remarkMath]}
  rehypePlugins={[rehypeKatex]}
  components={{
    code({ node, inline, className, children, ...props }) {
      const match = /language-(\w+)/.exec(className || '');
      return !inline && match ? (
        <SyntaxHighlighter
          {...props}
          style={irBlack}
          language={match[1]}
          PreTag="div"
        >
          {String(children).replace(/\n$/, '')}
        </SyntaxHighlighter>
      ) : (
        <code {...props} className={className}>
          {children}
        </code>
      );
    },
  }}
>
  {post.content}
</ReactMarkdown>
```

To style everything nicely, I used a dedicated stylesheet with Tailwind `@apply`
directives. I think it came together quite well. Also, here's some math:

$$
\begin{align*}
    D\int_\reals x^4\,u_{xx} \; dx &= -4\,D \int_\reals x^3\,u_{x} \; dx \\
                                  &= 12\,D \int_\reals x^2\,u \; dx
\end{align*}
$$


### Hosting
My domain *samkwort.com* was previously hosted by Google Domains, which was
recently sold to Squarespace. As such, I decided to migrate my domain to
Cloudflare after reading some very positive reviews online.

Like my previous site, hosting is done through GitHub Pages. I used the
[Deploy to GitHub Pages Action][4].

## Conclusion
JavaScript is weird. I definitely prefer coding in C and Python, but it was good
to branch out a little bit. I'm a big fan of the new site; making content is
super straightforward, and the design is a big upgrade.

[1]:https://nextjs.org/docs/pages/building-your-application/routing/dynamic-routes
[2]:https://github.com/remarkjs/react-markdown
[3]:https://github.com/vercel/next.js/tree/canary/examples/github-pages
[4]:https://github.com/marketplace/actions/deploy-to-github-pages







A New Website


As part of my current role at work, I'm required to automate the production of
many plots using results from PSCAD simulations. 
These plots are generated by `matplotlib`.
To streamline the plot reviewing process, we generally combine the plots into
a single PDF file. Currently, this process is incredibly slow, sometimes taking
minutes to generate a single PDF, with 50+ of pages.
As a learning exercise in Python benchmarking, and to improve performance, I
decided to undertake an investigation into this component of our simulation
workflow, hopefully removing the bottleneck.

## Current Practices
The current methodology for generating PDFs looks something like this:

1. Generate simulation data for each scenario, including plots.
2. Store folders names containing output plots in dictionary by 
   scenario number.
3. Iterate through output folders using dictionary and generate PDF
   for each scenario.

I narrowed down the performance bottleneck to step 3 above.
For this step, we have the following Python code:

```python
# Create unique pdf for each scenario
for key in pdf_groups.keys():
    pdf = FPDF(orientation='L', unit='mm', format='A4')
    
    # Loop through test in scenario, keeping track of first and last
    first_test = ""
    last_test = ""
    for test in pdf_groups[key]:
        if last_test == "":
            first_test = test

        for signal in SIGNALS_TO_CAPTURE:
            image = f"{out_folder}\\{test}\\{test}_{signal}.png"
            # Only add the graph to PDF if it exists.
            if (os.path.isfile(image)):
                pdf.add_page()
                pdf.image(image, x=0, y=20, w=300, h=150)
            else:
                print(f"File: {image} does not exist.", file=sys.stderr)

        last_test = test

pdf.output(out_folder + "\\" + first_test + "-" 
            + (re.search(r"[0-9]+",last_test)).group(0) + ".pdf", "F")
```

The code itself is $O(n\times m)$, where $n$ is the number of scenarios
and $m$ is the number of tests.

My intuition tells me that the bottleneck is somewhere in the `FPDF` package,
which is used here to embed the images in generated PDF output. We will
benchmark this entire code segment.

## Benchmarking
There are a few options for benchmarking the speed of code in Python. Some of
these are:

- [`timeit`][1]
- [`line_profiler`][2]
- [`pyperf`][3]

I'm opting to use `line_profiler` as this will give me a line by line breakdown
of my code's execution speed.

### Benchmarking Setup
To support the profiling process, we need to isolate this segment of code and
create a custom script for profiling. Here's what I ended up with.
```python
@profile
def benchmark_with_FPDF(tests, out_folder, in_folder):
    pdf = FPDF(orientation='L', unit='mm', format='A4')
    for test in tests:
        for signal in SIGNALS_TO_CAPTURE:
            image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
            # Only add the graph to PDF if it exists.
            if (os.path.isfile(image)):
                pdf.add_page()
                pdf.image(image, x=0, y=20, w=300, h=150)
            else:
                print("File: " + image + " does not exist.")


    pdf.output(out_folder + "\\FPDF_benchmark.pdf", "F")
```

Note the use of the `@profile` decorator; `line_profiler` uses this to signal
which function to profile.

To profile the script with `line_profiler`, we just run:
```
kernprof -lv script_to_profile.py
```

### Results
Before using line_profiler, I ran a very crude test shown below.
```python
start_time = datetime.now()
benchmark_with_FPDF(tests, out_folder, in_folder)
total_time = datetime.now() - start_time
print(f"Benchmark took {str(total_time)}")
```
This gave me a runtime of `45.574701 seconds`. Note that the input was 24 PNG
images, generating a 24 page PDF.

The results from the intial profiling are captured below.
```python
Benchmark took 0:01:48.941374
Wrote profile results to benchmark.py.lprof
Timer unit: 1e-06 s

Total time: 108.941 s
File: .\benchmark.py
Function: benchmark_with_FPDF at line 20

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    20                                           @profile
    21                                           def benchmark_with_FPDF(tests, out_folder, in_folder):
    22         1         59.7     59.7      0.0      pdf = FPDF(orientation='L', unit='mm', format='A4')
    23         3          1.6      0.5      0.0      for test in tests:
    24        24         25.5      1.1      0.0          for signal in SIGNALS_TO_CAPTURE:
    25        24         39.2      1.6      0.0              image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
    26                                                       # Only add the graph to PDF if it exists.
    27        24       2324.0     96.8      0.0              if (os.path.isfile(image)):
    28        24        708.6     29.5      0.0                  pdf.add_page()
    29        24  108695906.0 4528996.1     99.8                  pdf.image(image, x=0, y=20, w=300, h=150)
    30                                                       else:
    31                                                           print("File: " + image + " does not exist.")
    32
    33
    34         1     242092.4 242092.4      0.2      pdf.output(out_folder + "\\FPDF_benchmark.pdf", "F")

```

According to the results, 99.8% of execution time is spent on this FPDF API
call.
```py
pdf.image(image, x=0, y=20, w=300, h=150)
```
My intuition seems to be correct; it's clear that the bottleneck is in FPDF.

## Alternatives to FPDF
After some research, I found the following tools to be candidates for replacing
FPDF.

- [`img2pdf`][4]
- [`Pillow`][5]

It seems that `img2pdf` actually uses `Pillow` internally. These `img2pdf` API
seems straightforward to use, so I'll just be testing that. 

The benchmarking function rewritten to use `img2pdf` is included below.
```python
@profile
def benchmark_with_img2pdf(tests, out_folder, in_folder):
    # Get a list of paths to the images
    paths = []
    for test in tests:
        for signal in SIGNALS_TO_CAPTURE:
            image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
            # Only add the graph to PDF if it exists.
            if (os.path.isfile(image)):
                paths.append(image)
            else:
                print("File: " + image + " does not exist.")

    # Specify paper size (A4 landscape)
    a4inpt = (img2pdf.mm_to_pt(297), img2pdf.mm_to_pt(210))
    layout_fun = img2pdf.get_layout_fun(a4inpt)

    # Generate PDF
    with open(f"{out_folder}\\benchmark_img2pdf.pdf","wb") as f:
        f.write(img2pdf.convert(paths, layout_fun=layout_fun))
```

As before, we run the script again with `line_profile`.

### Results
Using my crude test script again, the img2pdf API only took 1.887416 seconds.
The `line_profiler` results confirm this:
```python
Total time: 94.2377 s
File: .\benchmark.py
Function: benchmark_with_FPDF at line 21

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    21                                           @profile
    22                                           def benchmark_with_FPDF(tests, out_folder, in_folder):
    23         1        110.0    110.0      0.0      pdf = FPDF(orientation='L', unit='mm', format='A4')
    24         3          2.5      0.8      0.0      for test in tests:
    25        24         44.1      1.8      0.0          for signal in SIGNALS_TO_CAPTURE:
    26        24         63.1      2.6      0.0              image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
    27                                                       # Only add the graph to PDF if it exists.
    28        24       3338.2    139.1      0.0              if (os.path.isfile(image)):
    29        24       1062.9     44.3      0.0                  pdf.add_page()
    30        24   93822206.6 3909258.6     99.6                  pdf.image(image, x=0, y=20, w=300, h=150)
    31                                                       else:
    32                                                           print("File: " + image + " does not exist.")
    33
    34
    35         1     410847.1 410847.1      0.4      pdf.output(out_folder + "\\FPDF_benchmark.pdf", "F")

Total time: 1.88704 s
File: .\benchmark.py
Function: benchmark_with_img2pdf at line 37

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    37                                           @profile
    38                                           def benchmark_with_img2pdf(tests, out_folder, in_folder):
    39                                               # Get a list of paths to the images
    40         1          0.5      0.5      0.0      paths = []
    41         3          1.2      0.4      0.0      for test in tests:
    42        24          8.7      0.4      0.0          for signal in SIGNALS_TO_CAPTURE:
    43        24         25.2      1.1      0.0              image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
    44                                                       # Only add the graph to PDF if it exists.
    45        24        938.6     39.1      0.0              if (os.path.isfile(image)):
    46        24         15.7      0.7      0.0                  paths.append(image)
    47                                                       else:
    48                                                           print("File: " + image + " does not exist.")
    49
    50                                               # specify paper size (A4 landscape)
    51         1          3.9      3.9      0.0      a4inpt = (img2pdf.mm_to_pt(297), img2pdf.mm_to_pt(210))
    52         1          9.8      9.8      0.0      layout_fun = img2pdf.get_layout_fun(a4inpt)
    53
    54                                               # Generate PDF
    55         1        408.9    408.9      0.0      with open(f"{out_folder}\\benchmark_img2pdf.pdf","wb") as f:
    56         1    1885623.0 1885623.0     99.9          f.write(img2pdf.convert(paths, layout_fun=layout_fun))
```

- FPDF took approximately 94.2377 seconds to complete the conversion process.
- img2pdf completed the same task in approximately 1.88704 seconds.

Therefore, by migrating our image to PDF conversion scripts to use img2pdf, we
should observe a 50x increase in PDF generations speeds. This is huge for
multi-hour batched simulations runs, which will have 50+ test cases across
multiple scenarios.

## Conclusion
There was a major bottleneck in our image to PDF conversion code. By moving
from `FPDF` to `img2pdf`, we saw a 50x increase in program execution speed.


[1]:https://docs.python.org/3/library/timeit.html
[2]:https://github.com/pyutils/line_profiler
[3]:https://github.com/psf/pyperf
[4]:https://github.com/myollie/img2pdf
[5]:https://pillow.readthedocs.io/en/stable/

Sam Kwort

All Posts