Running Lightning on Pruned Bitcoin Nodes – Jason Comeau
The primary audience of this article is technical people who have already set up and have run bitcoin and lightning nodes before.
If are thinking of a way to build long-lasting and robust deployments for an application you are making or want to power a large lightning node farm, please read on.
Also, before you comment below, it is possible to run Lightning (both LND and c-lightning) with
bitcoind pruned mode.
This article will not include “script commands”, I assume you know how to run and manage a node in general. If you need scripts and commands, I can write a separate article for that topic.
One thing I must stress is you must learn how to run and manage pruned nodes if you hope to run infrastructure for many years. Only running full node implementations are ok for now, but will become impossible to maintain over the coming years, due to ever-increasing hardware requirements for the ever-increasing blockchain.
In my travels, I noticed, there are not enough publications talking about this topic. So I wanted to write a quick article just articulating some of these risks involved in infrastructure and how you can overcome them, and to just start the conversation.
To help you scale, you must start thinking about deploying a multi-tiered full node and pruned node architecture.
No hands are held, and you can lose lots of money very quickly, so please plan this out carefully.
I would classify Lightning as still “experimental” phase, but the below tips can help you reduce the risk of losing your money or experiencing outages.
Bitcoin pruned nodes also come with a high level of complexity and risk for downtime, as the only way to recover from a failed pruned node is a complete re-sync of the blockchain and a re-pruning. This means a single node could be down and out of action for days if you don’t plan accordingly.
This article will help you quantify and reduce your risks, so you don’t lose money or experience delayed downtime, and have a terrible experience with bitcoin and lightning nodes.
Of course, you can run your nodes on containers and servers in public clouds, but I do not advise this if you have a significant amount of money on your nodes. It is trivial to steal your lightning and bitcoin money on infrastructure that is not adequately secured.
If you want to run infrastructure with any significant amount of money on it, this needs to be private and secured equipment.
You need to realise; we have an ever-expanding blockchain, this means your hardware and network requirements will still be going up over the years. You need to have a plan to handle this effectively if you are serious about running blockchain infrastructure you can’t leave this to fate.
Two main areas Bitcoin Full Node uses resources are:
1. Disk space
2. Network Bandwidth
Some top ways to limit and manage your infrastructure usage for Bitcoin nodes are:
Full Bitcoin Node Tips:
- Use full nodes as a way to store master data closer to your pruned nodes; this way, it is faster to recover from a failure of a pruned node.
- Limit the number of network connections to your full node, to save on bandwidth costs; you only need a few to keep the blockchain in sync. Full nodes can use crazy amounts of bandwidth if left unchecked.
- Keep the blockchain data on slower disks like SAS drives, don’t bother mirroring this data, if you lose this data you need to re-synch with the network, so no point spending money on data loss prevention, it’s impossible to lose the blockchain, never waste money on backing this up!
- You should only need four cores and four gigs of ram if you have limited your network connections above.
- 100MB internet pipe should be enough, but you need an internet pipe that you can throttle up to 1GB if you need to re-synch the blockchain quickly (SD-WAN or Mega Port is excellent for this)
- IF you run a hot wallet, then encrypt it, also back this up to another location every 4 hours in an encrypted state.
Pruned Bitcoin Node Tips:
When you run second layer solutions like Lightning, you can run them on pruned nodes, but you need to make sure block heights stay in synch. You can use the CLI tool of bitcoin and Lightning to make sure blocks remain in sync, some scripting required here.
I think running lighting on pruned nodes is the only real way to go for any long term deployment lasting many years as the blockchain requirements increase rapidly.
Some tips for running Lightning ontop of Pruned Bitcoin nodes:
- Set node to pruned mode, but make sure you don’t make it too small.
- You want a decent pruned horizon time, so set pruned size to a size that gives you time to bring down your lightning node and bring it back up again within the pruned horizon.
- Run your lightning-pruned nodes on fast equipment, with SSD storage, to make sure they are always in synch. A pruned node has more disk IO activity and uses more CPU.
- Pruned bitcoin directory should be stored on mirrored drive set that is separate to the lightning directory; this helps you recover quicker if your disks fail.
- If your bitcoin RPC connection goes down, your lightning node will crash as well so converge the Lightning and Bitcoin infrastructure stack and run on the same instance, that way you lower your latency between your lighting node and bitcoin node and reduce the risk of sync failure and eclipsing the prune horizon.
- Lightning node backup and restore is experimental at best and very risky, so always keep your lightning directory mirrored fault-tolerant hardware.
- Drain old nodes, then create new nodes, never migrate! It is way too risky.
How your lightning node can go past the “prune horizon.”
- If your lightning daemon is off and is stuck on a particular block height
- bitcoin node prunes blocks past the block height your lightning node requires to catchup
- Lightning node turns back on, and can’t find the pruned block and is now forever stuck.
- Network failure between your lightning node and bitcoin node
- Corrupt blockchain on either side
- prolonged disk failure on either side
An excellent way to monitor this situation is to use a block height check on your lightning node and your bitcoin node. Ideally, these should be equal in size.
Also, for the sake of planning, you can use your desired outage windows to help you calculate how much prune size you need.
This is not an exact calculation as the block sizes fluctuate, but helps you set a select prune size with some logic other than just guessing.
Below is the calculation I use to plan how much disk space I need for a pruned bitcoin node with a good prune horizon time:
outage window desired = one week
prune horizon desired = two weeks
new block generated every 10 minutes = 144 blocks a day
disk space needed = 144 blocks * 1MB = 144mb a day growth rate, of 1GB for a week.
prune horizon = ( 2016 blocks) * 1MB (average block size) for two weeks of blocks = 2016MB
So for a week, that means you need a minimum of 1GB of disk space required for pruning, and because I am an engineer I live by the multiply by three rule.
So that means I set my prune to at least 3GB, this will give me a prune horizon of two weeks at least.
If you do go over your prune horizon and your lightning node is stuck, you can fix this situation by:
- Remember your lightning node can use ANY bitcoin data source as long as it knows the RPC user name and password.. useful thing to know if your pruned bitcoin node fails.
- Shutdown the failed, pruned bitcoin node daemon
- Use SDWAN or on-demand network links (ssh tunnel or VPN, etc.) to connect your failed lightning node, to an RPC port of your full node when you need to “catchup” your lighting node.
- After everything is caught up shut down the lightning daemon
- Do a full re-sync of the failed pruned node by turning off prune mode
- Turn prune mode back on
- Turn lightning daemon back on
As you can see, you can experience significant downtime if your prune node fails and you go beyond your prune horizon.
If you want to setup monitoring, use the below equation as a guide to track your sync rate between your lightning nodes and your master bitcoin node.
Monitoring Equation = (Bitcoin Block Height — Lightning Node Block Height) > 0 = warning
Monitoring Equation = (Bitcoin Block Height — Lightning Node Block Height) > 1000 block delta = extreme warning
Monitoring Equation = (Bitcoin Block Height — Lightning Node Block Height) > 2000 block delta = critical shutdown to prevent eclipse of prune horizon
The main reason to have a multi-tiered infrastructure and have an up to date full node on-premise is so you can quickly re-sync when you need to.
Also, if you set your prune size large enough, this gives you time before you fall over the prune horizon.
Now you might say to yourself; this looks like a lot of work, why don’t I make my life easier and run full nodes.
Well, for now, yes you can do that, but you must realise with a growth rate of 144MB a day (assuming no block size changes), the blockchain will be an unmanageable size very shortly.
My firm belief is the blockchain size is unlikely to get smaller, so this means baring any breakthrough in node size, I think it is prudent to start learning how to run lightning nodes ontop of pruned bitcoin nodes now.
A picture is worth a thousand words, so please see below, a diagram of the infrastructure layout I have been talking about in this article.
This multi-tiered approach allows you to scale your lightning nodes very quickly with automated scripting.
I have not experimented yet with load balancing configuration, this could further strengthen and simplify the design, but I will update this article as I experiment.
Another thing that can be done is containerisation of your lightning nodes and pruned bitcoin nodes; you can have separate docker or kubernetes containers linked using SDWAN of each node and theoretically scale as fast as required.
The only issue holding this back is automated channel management and balancing, which is very difficult right now.
Lightning will become more adopted once people develop fully scalable and reliable lightning infrastructure and cloud systems that abstract all the complexities away from the users.
I love lightning technology, and it does indeed work. I don’t see Lightning widely adopted until we build out a cloud solution that abstract the complexity away from all of us.
I have learned it is very complicated and risky running a lightning node, and there needs to be a converged architecture that includes bitcoin and Lightning in the one package.
The issue with layer two solutions is that you add complexity to your design; some may argue it would be wiser to include lightning functionality into the bitcoin protocol itself. However, I don’t want to start another debate, but from an infrastructure and complexity perspective, you can see a layer two solution adds significant risk and cost to running infrastructure solutions.
Bitcoin and Lightning are lead by developers; you can see that in the way they design the systems, it is very developer-centric and has lots of complexity.
We need to start introducing infrastructure specialists and scientists in the development of bitcoin and Lightning, so we get a balanced approach and a simpler architecture that is supportable and sustainable well into the future.
Do you need commands?
IF there is anything in this article you want commands for, let me know, I am more then happy to do a follow up with actual commands in them to help people learn.