A bit more context, as I’ve been doing some performance characterization and optimization in spare time…
Notes (these numbers correspond to markings on the screenshot):
- When a new block is created, indexing kicks in. That’s a write-heavy activity. You can see at the same time there’s no significant network traffic - it’s all internal stuff.
- In order to create indexes, Bitcoin Core must read its data, so there’s a lot of reading going on too.
- You can’t expect more than 150-200 IO (combined reads + writes) from a HDD so when you get over 150, you can see timeouts. As you can see here, it goes to 250 sometimes. But the entire time in large rectangles was spent in Counterparty Server RPC timeouts. (Edit: 1 row = 1 second; so Counterparty Server didn’t quit .)
Note that Counterparty Server runs on a different system, which makes performance troubleshooting easier. Its performance in this case isn’t an issue (although it too would benefit from SSD) so I’ll just show another screenshot from Bitcoin Server when times are good (i.e. no timeouts on RPC client):
- Plenty of reads, because Counterparty Server is catching up after timeouts
Plenty of writes, as well, but only 15-20 per second, vs. 20-40 before (the second screenshot was taken 5 mins after the first).
- You can see bursts of 2 MB/s going out to the network.
PktIn is regular Bitcoin network and whatnot incoming traffic, not different from the first screenshot .
These outgoing batches of data is what Bitcoin Core sends to RPC client (Counterparty Server). The size of these can vary and at least partially depends on rpc-batch-size = in Counterparty’s server.conf. In this particular case I have rpc-batch-size = 500 which seems a bit too much for this particular setup (i.e. slow HDD on the system running Bitcoin Core). The default is 20 and it seems 200 was causing fewer timeouts during busy times on Bitcoin Core. When I used rpc-batch-size = 100 then batches of network sends were smaller (as one might expect). And in bitcoin.conf you can also rpcthreads and rpctimeout (the first one is pointless to increase if your server can’t take it; the second one maybe can allow the server to tolerate longer timeouts).
I also looked into the internals of Bitcoin Core and Counterparty Server performance in terms of I/O and while I haven’t processed most of data I collected, I seems to me that LevelDB (txindex, specifically) is what causes a lot of IO on Bitcoin Core, and haven’t found a way to make it run faster. Apart from the obvious, of course, which is to increase dbcache to a higher value if you have more RAM.
Realistically, though, you need SSD for Bitcoin Core and if you have more than 4 GB of RAM then you can put Counterparty Server on the same system. Below that and/or without SSD, it’s tough.
- Bitcoin Core and Counterparty Server on the same system (screenshot below) can increase bandwidth consumption
- More importantly, though, one doesn’t have to wait long enough to see (Reads + Writes) hit 200 or even more (over 300, in the screenshot, and I waited just 2-3 minutes). At this point without SSD you’d likely get I/O timeouts in Counterparty and your app. Maybe not immediately as this was just a spiike, but pretty soon.
With SSD you can go to few thousand IOPS and not only avoid timeouts, but also recover from them (if they happen) without Counterparty Server timing out 12 times and then quitting (stopping).
When the blockchain gets 2-3 new blocks within 15 mins, or a 2-3 block reorganization (similar effect, essentially) a lot of churning has to be done, and a single HDD can’t handle that in a timely manner and still respond to RPC requests by Counterparty Server.
##Mempool Cache
You can see you loaded 20,000 transactions from the mempool cache, that’s a lot. In July 2015 (July 2015 flood attack - Bitcoin Wiki) the attack loaded 80K.
At time of attack you can’t do much even with faster h/w. But at least have this in bitcoin.conf:
minrelaytxfee = 0.00005
limitfreerelay = 0
In Bitcoin Core 0.12 (you can get a release candidate with addrindex from BTCDrak’s repo now - I’ve been running it for a few days, it seems good enough for test/dev) this will be handled better, and these settings won’t be necessary - see Reducing bitcoind memory usage · GitHub.
