Paul Cavallaro's Blog on Paul Cavallaro

tcmalloc's Temeraire: A Hugepage-Aware Allocator

Fri, 25 Jul 2025 10:00:00 -0500

Today’s paper is Beyond malloc efficiency to fleet efficiency: a hugepage-aware memory allocator by A.H. Hunter, Chris Kennelly, Paul Turner, Darryl Gove, Tipp Moseley, and Parthasarathy Ranganathan from OSDI ‘21.

This paper is a great read, less for its key results – a hugepage aware allocator that reduces tlb misses and memory usage – but for its meta-lessons: measure the entire system, invest in tooling, build a process to experiment confidently, profile globally act locally.

2022 Predictions

Sun, 02 Jan 2022 00:30:00 -0400

Inspired by others who have written predictions in order to keep themselves accountable and to improve their predictive abilities, here are some predictions I have for 2022.

They roughly fall into a few categories: cryptocurrency, technology, long-term technology, and fun (i.e. sports).

Cryptocurrency Predictions

80% certain that Bitcoin price will end the year greater than $10,000, and less than $150,000.
80% certain that Ethereum price will end the year greater than $500, and less than $10,000.
80% certain that Solana price will end the year greater than $20, and less than $1,000.

These predictions are all part of my thesis that cryptocurrency will continue to exist for at least the rest of my lifetime as a new asset class for believers, grifters, and speculators. Only outlawing or strict regulation could really change this prediction. My basic theses is that as technologists have had increasingly larger holding of wealth, the tech subculture that enjoys crypto can in prepetuity continue to pump money into it. Another theory I only slightly believe is that as the world economy continues to grow, I predict there will be more and more fragmented subcultures that can be propelled forever by their own true believers and the FOMO of others. A trillion dollars really isn’t what it used to be anymore.

The Deadweight Loss of "The Deadweight Loss of Christmas"

Mon, 06 Dec 2021 10:00:00 -0400

Every year as the weather chills, wreaths are hung, hot cocoa is sipped, and stores start playing the sweet dulcet tones of George Michael and Mariah Carey, our thoughts come to focus on Christmas. And like all good analytical worker bees, inevitably someone asks why must we buy presents for others, instead of everyone just buying whatever it is that they want. It’s a reasonable question! It’s tough to know what your father-in-law wants, and you really can only buy so many scarves for your mother before everyone catches on that you just can’t think of anything to get her.

Facebook's Tectonic Filesystem:
Efficiency from Exascale

Sun, 25 Jul 2021 10:00:00 -0400

Today’s paper is Facebook’s Tectonic Filesystem: Efficiency from Exascale from FaST ‘21. The paper covers the Tectonic Filesystem at Facebook, its implementation, and various design decisions they made. I’ll summarize the paper and do a deeper dive on some of its highlights.

While there is a rich history of filesystem papers and systems, it’s nice to read this Tectonic paper, since many of the oft-quoted papers are getting a bit long in the tooth. The Google File System (2003) paper is almost two decades old but is cited more than Bill Murray in the wild.¹ There are some more modern papers, mostly from Microsoft like Windows Azure Storage (2011) and Azure Data Lake Store (2017), but it’s nice to see something from a different company.

Posix Permissions Are Weird

Sat, 13 Mar 2021 10:00:00 -0400

I’ve been working on a distributed filesystem recently, which has exposed me to the POSIX permission system in more depth, and whoo, I’m convinced that POSIX permissions are weird and bad and can lead to some very surprising¹ behavior.

Deleting Files

Quick, tell me what permissions you need to have to delete the file at /home/paul/notes.txt ?

Well, the surprise here is that you need absolutely zero permissions on the file itself. All you need are:

Fanouts and Percentiles

Sun, 26 Jul 2020 12:00:00 -0500

In today’s post we’re going to talk about latencies in a common distributed system architecture: The root-leaf, or parent-child, fanout architecture. We’ll try to gain an intuition for what drives tail latency, derive a formula for tying together the parent and child latency distributions, and provide a useful interactive visualization to help understand tail latency in these systems.

Root-Leaf Fanout Architecture

When dealing with distributed systems, it’s a common pattern to have a root aggregator node that fans-out requests to many leaf nodes that do work in parallel, then send back their response to the root node to aggregate and create a final response. Usually the API of the system is exposed through the root-aggregator nodes, so the latency of these root nodes is what matters.

Common Systems Programming Optimizations & Tricks

Mon, 26 Aug 2019 10:00:00 -0400

Today’s blog post is an overview of some common optimization techniques and neat tricks for doing “systems programming” – whatever that means today. We’ll walk through some methods to make your code run faster, be more efficient, and to squeeze just a little more juice from whatever you got.

All of the examples we’ll go over are also on github at paulcavallaro/systems-programming.

False sharing is a fairly well-understood problem in optimizing multi-threaded code on modern SMP systems. The problem has been written about fairly extensively, but the basic idea is that physical memory on a machine isn’t infinitely granular, i.e. you can’t just read a byte.¹ Instead, when you want to read a single byte of memory, the processor will pull in and cache not just that byte, but the surrounding data as well, on the assumption that it will likely be used too. This unit of data that gets read and cached is called a “cache line”, and is essentially the smallest piece of memory that can be accessed.

A Practical Multi-Word Compare-and-Swap Operation

Sun, 17 Jun 2018 12:00:00 -0400

Today’s paper is about how to implement an efficient and practical multi-word¹ compare-and-swap operation. The paper is entitled “A Practical Multi-Word Compare-and-Swap Operation” by Timothy L. Harris, Keir Fraser, and Ian A. Pratt constructs a lock-free, non-blocking, multi-word CAS given just a single word CAS.

Since the focus is on practicality, the authors write up their algorithms in almost-real pseudo C code, with a minimum of hand-waving. As an example, here is their definition of CAS1, assuming the entire function operates atomically.

x86 TSO: A Programmer's Model for x86 Multiprocessors

Sun, 28 Jan 2018 17:00:00 -0500

The paper I’m writing about today is x86-TSO: A Rigorous and Usable Programmer’s Model for x86 Multiprocessors by Sewell, Sarkar, Owens, Nardelli, and Myreen. The paper tries to provide a framework for systems programmers to reason about the execution of code on x86 multiprocessors. The main focus is being the execution of multi-threaded code on multi-processors, and how we can reason about its many possible executions.

Prior to learning about x86 processors, I had assumed (like most?) that non-synchronized code execution proceeded as an arbitrary interleaving of assembly instructions, where writes from one thread were immediately readable by all others. This still leaves lots of possible total execution orderings, but unfortunately this view is even too simplified.

New Year, New Resolution

Sun, 21 Jan 2018 21:54:15 -0500

So I had a resolution last year to read and write a blog post about 26 papers in 52 weeks. I managed to write about 5 papers in 2017, which isn’t 26, but also isn’t 0. I’m chalking that up as a win, and I definitely ended up reading more interesting papers last year than I had previously.

12 Months, 12 Papers

So for 2018 I’m aiming for the more manageable, but certainly aspirational, 12 papers in 12 months. My next post will be on a cool paper that presents a programming model for how x86 multiprocessors work. So if you’ve ever broken out into a cold sweat about what guarantees there are from your hardware when writing or debugging multi-threaded code, then this is the post for you.

System Software

Sun, 17 Dec 2017 22:33:25 -0500

There’s a quote I love from Ian Lance Taylor’s series on linkers about the complexity that creeps into them.

As we’ve seen, what linkers do is basically quite simple, but the details can get complicated. The complexity is because smart programmers can see small optimizations to speed up their programs a little bit, and somtimes the only place those optimizations can be implemented is the linker. Each such optimizations makes the linker a little more complicated. At the same time, of course, the linker has to run as fast as possible, since nobody wants to sit around waiting for it to finish. Today I’ll talk about a classic small optimization implemented by the linker.

Teams, Communication, and Code

Tue, 10 Oct 2017 12:30:00 -0400

It seems almost an accepted wisdom in Software Engineering today that small, empowered, cross-functional teams are the best way to build software and products. You can’t have design be separate from product be separate from engineering. Good products are about making the correct choice on hundreds, if not thousands, of tradeoffs. If you solve for your silo’s perceived constraints, versus the actual constraints, you won’t make the correct choice.

I’d posit that some recent Software Engineering concepts have risen out from this insight: mainly Microservices. The biggest improvement Microservices promises is not cleaner code, or ease of development (I’d argue that it hurts both), but that it allows individual teams to make progress on their product independently of other teams. Microservices enables us to actually make progress in a world where nobody knows the whole system, but everybody knows some small piece.

Optimizing Function Placement with Call-Chain Clustering

Sun, 09 Jul 2017 12:00:00 -0400

Today’s paper is from Facebook’s HHVM team: Optimizing Function Placement for Large-Scale Data-Center Applications. The paper describes two ways to improve code layout, aimed specifically at very large applications.

The first method is to optimize function layout, i.e., where functions, after inlining is performed, are placed inside the binary. If function A calls function B many times, we’d want B to be placed near A so that it’s more likely that the instructions for B will already be in the instruction cache.

The Bw-Tree

Sun, 25 Jun 2017 12:00:00 -0400

Today’s paper is The Bw-Tree: A B-tree for New Hardware Platforms. The paper describes the Buzzword Tree, a new B-tree variant. B-trees and their modern successor the B+ trees are heavily used inside DBMSes for indexes, making them performance critical.

The Bw-tree focuses on improving performance on current and future hardware. For the Bw-tree this means optimizing for multi-core scalability, cpu cache usage, and flash storage. This paper in particular focuses on the novel techniques used to optimize the in-memory aspects of the Bw-tree, largely ignoring the flash storage layer.

Hashed and Hierarchical Timing Wheels

Tue, 28 Feb 2017 17:25:23 -0500

Today’s paper was written in 1987 by George Varghese and Tony Lauck from Digital Equipment Corporation (!), and has withstood the test of time. It’s about how to efficiently implement a timer facility that allows you to start a fixed length timer and perform some action once it has expired. The ability to set timeouts on requests or operations is almost a given, so being able to handle these timers efficiently is quite important.

Flexible Paxos

Sun, 29 Jan 2017 23:15:00 -0500

Today we’re going to talk about a new development in the world of consensus protocols called Flexible Paxos from Heidi Howard, Dahlia Malkhi, and Alexander Spiegelman. Flexible Paxos builds off of Paxos, which is the foundational consensus protocol invented by Leslie Lamport and has been written about extensively. Paxos is notoriously difficult to understand and reason about, so much so that there is a follow up paper by Leslie Lamport called Paxos Made Simple which, while still being easier to understand than the original paper, is still difficult to follow. There has been other work on consensus protocols like Zab which powers ZooKeeper, and more recently Raft which powers Consul, etcd, and others. That said, Paxos is still the grandaddy of them all, and powers lots of important infrastructure.

Write-Behind Logging

Sun, 15 Jan 2017 10:12:00 -0500

This paper by Joy Arulraj, Matthew Perron, and Andy Pavlo describes a new way to rethink the DBMS Write Ahead Log (WAL) for NVM technology.

NVM stands for Non Volatile Memory which means that it won’t lose the data written to it when it loses power, unlike DRAM. The reason NVM is such a big deal is that it promises to provide read/write latency around 2-8x of DRAM, have good random read/write latency, and have much larger storage capacity than DRAM. Compared to NAND flash Intel claims their new 3D XPoint will be 10x lower latency, 4x write throughput, 3x read throughput.

New Year's Resolution

Wed, 04 Jan 2017 22:59:21 -0500

52 Weeks, 26 Papers

This year I had a few New Year’s resolutions, and one of them was to read and write up a short summary of one academic paper every two weeks.

If I succeed, by the end of 2017 I’ll have blog posts covering twenty six different papers. I’d say that’d be a pretty successful year. We’ll see how it turns out.