raw_transactions_cache_size in counterparty-lib

Support Server
1

If counterparty-server runs in verbose mode, this can be seen in the log:

[DEBUG] getrawtransaction_batch: txhash_list size: 452 / raw_transactions_cache size: 20000 / # getrawtransaction calls: 452

The other day I noticed this in another post related to the long startup time, so today I checked this in counterparty-lib’s source code.

Bitcoin Core

When the back-end server is Bitcoin Core, the value is 20,000.

I don’t understand the technical side of this well enough (so feel free to comment and correct), but if transaction size is around 500 bytes each, 20,000 means around 10 MB of RAM that counterparty-server would dedicate for this.

In Bitcoin Core 0.12 there’s a new configuration value maxmempool (MB) (default: 300), so it would be interesting to know if maxmempool=20 and BACKEND_RAW_TRANSACTIONS_CACHE_SIZE=30000 would both save RAM and improve the performance.

And what are disadvantages of using maxmempool=10, if any, while retaining the default setting in counterparty-lib.

I suppose maxmempool could be set to a much smaller value (such as 10 MB) if BACKEND_RAW_TRANSACTIONS_CACHE_SIZE=20000. Otherwise, during spam attacks such as the one that happened 2 days ago, Bitcoin Core bloats its mempool for nothing (spam transactions) while counterparty-server can’t cache those transactions because it’s limited to 20,000.

rpcthreads

(This is off-topic but I don’t want to create another post just for this parameter.)

Interestingly, right below that transactions cache setting there’s BACKEND_RPC_BATCH_NUM_WORKERS which I suppose should equal the number of RPC threads in Bitcoin Core. But in Counterparty Server the default value is 6, and in Bitcoin Core it’s 4:

-rpcthreads=<n>
       Set the number of threads to service RPC calls (default: 4)

On the other hand, for some reason at one point in the past it appeared that a far larger number of threads is beneficial, so the official Counterparty documentation for Bitcoin Core recommends (and Federated Node defaults to) rpcthreds=1000.

I can’t remember what was the reason for this ludicrous value (it’s somewhere in counterparty-lib Github issues), but should probably be revisited. I set my rpcthreads to 4 (number of CPU cores). Maybe counterparty-server wouldn’t have as many timeouts if I set it to a higher value, but in the past I had this set to 100+ and I still had problems.

btcd

btcd isn’t currently supported because it doesn’t support JSON 2.0, but the value can be set here and it’s 10,000:

Once btcd supports JSON 2.0 (for RPC request batching) this should probably be changed.

2

afaik that cache is always full, because its not just for 0conf

           block = backend.getblock(block_hash)
           previous_block_hash = bitcoinlib.core.b2lx(block.hashPrevBlock)
           block_time = block.nTime
           txhash_list = backend.get_txhash_list(block)
           raw_transactions = backend.getrawtransaction_batch(txhash_list)

thats the block updating stuff
which is unnecesary, we have the full block in block so we could use the raw TXs from there afaik

also 20k * 500 bytes is indeed not that much
I think in practice it’s consuming more memory, but let me debug that a bit and maybe we can set that limit a bit higher since it would improve performance a bit if we could without too much memory increase

also CP uses a flat fee, not estimated (which is a problem that could/should be worked on I guess), so setting a low maxmempool will probably result in CP txs getting dropped from ur mempool fairly quickly when stresstests happen