I decided to spend some time auditing Electrum-NMC’s bandwidth usage, to see if any optimizations were possible and/or needed. Using the excellent nethogs tool by Arnout Engelen, I determined that Electrum-NMC’s initial syncup downloads 672 MB, which is quite a lot. Clearly some optimizations would be highly welcome.

Upstream Electrum avoids downloading most of the Bitcoin blockchain’s headers via checkpoints. Basically, a checkpoint consists of a hardcoded block hash (and a difficulty target), specifically one for each difficulty retargeting period (2016 blocks). Electrum doesn’t typically download any headers that are between two checkpoints; instead it only downloads such headers if it needs to verify a transaction from one of those blocks. This reduces bandwidth usage considerably. Electrum’s debug console can generate a checkpoint file (assuming that you’re connected to a trustworthy ElectrumX server; I’m using my own ElectrumX server connected to my own Namecoin Core node, so my server is indeed trustworthy to me). Electrum-NMC hadn’t yet gotten around to setting any checkpoints, due to it being a lower priority than getting name transactions working. But now that name transactions pretty much work, I was able to try out checkpoints.

Interestingly, after I set checkpoints through about 38 kiloblocks ago (more on why I chose that below), Electrum-NMC started reporting blockchain validation errors, most of which seemed to be related to missing headers. After quite a bit of debugging, I figured out why this was happening: in Bitcoin, calculating the new difficulty target requires some information from the first and last block header for a given retargeting period. Electrum’s checkpoints supply this information. However, this is actually not mathematically correct behavior, and exposes the blockchain to what’s called a timewarp attack. A timewarp attack can only be successfully pulled off by an attacker who can already do a 51% attack, so the attack is difficult, but if it’s successfully done, the attacker can make the difficulty continually drop, thus allowing them to mint more coins than the expected subsidy per 2 weeks. Because a timewarp attack is difficult to pull off, and because fixing it constitutes a hardfork, Bitcoin never actually fixed that vulnerability. Namecoin, however, did fix it, at the same time that we hardforked to enable merged mining. How does the timewarp fix work? Well, it actually needs a block header from the previous retargeting period in order to calculate the new target. Guess what, Electrum’s checkpoints do not supply such information (which would be useless in Bitcoin). And thus, the target calculation code in Electrum-NMC was complaining that the header from the previous retargeting period wasn’t present.

After quite a bit of trying to find a way to solve the issue without highly invasive changes, I eventually stumbled on an easy hack. I did a one-line change of the code that reports the number of checkpoints available, so that if n checkpoints exist, Electrum-NMC calculates the block height to download under the impression that n-1 checkpoints exist. This means that Electrum-NMC will actually download the final checkpoint’s set of 2016 block headers before it moves on to the uncheckpointed headers. As a result, it actually will have the previous retargeting period’s headers available when it tries to calculate the new difficulty target. It’s a stupid hack, but it definitely seems to work.

So, with checkpoints set through 38 kiloblocks ago, I measured Electrum-NMC’s bandwidth usage with nethogs again, and this time it had dropped to 66 MB of downloaded data. Not bad!

But why did I set the last checkpoint at 38 kiloblocks ago? Well, if we’re doing name lookups, we always want to have block headers for the last 36 kiloblocks, since unexpired names could be anywhere within that range. As an experiment, I tried setting a checkpoint at circa 3 kiloblocks ago, and found that Electrum-NMC’s bandwidth usage dropped to 4.9 MB of downloaded data during initial syncup. However, trying to do a name lookup resulted in a “missing header” error. Not much of a surprise there. However, upstream Electrum handles this situation just fine when verifying currency transactions – if you try to verify a currency transaction that’s behind a checkpoint, Electrum just downloads the necessary headers before trying to verify it. I quickly figured out where that code was, and adapted it to work with the name_show console command. With this change, if a name is behind a checkpoint, the lookup succeeds – but if you haven’t already downloaded the relevant headers, Electrum-NMC downloads 2016 headers (about 3.2 MB according to nethogs) before verifying the name.

For now, I don’t intend to put checkpoints beyond 38 kiloblocks ago into Electrum-NMC, because downloading 3.2 MB to do a lookup adds noticeable delay. There are probably some ways this amount of data can be optimized significantly, but that’ll have to wait for another day. That said, 66 MB instead of 672 MB is pretty nice savings.

This work was funded by NLnet Foundation’s Internet Hardening Fund.