Reparsing all transactions occurs once or twice a day. Why does it happen? How can I avoid it? counterpartyd doesn’t accept any requests during the reparsing. It takes over 1 hours in my server spec (Ubuntu 4G memory). It is a very serious problem for production running.
2015-08-08-T02:27:51+0900 [WARNING] Blockchain reorganisation at block 368821.
2015-08-08-T02:27:51+0900 [INFO] Reparsing all transactions.
2015-08-08-T05:22:07+0900 [INFO] Database minor version number updated.
2015-08-08-T06:19:37+0900 [INFO] Send: 0.00949802 BITCOINEX from 1GoVbdSUuKBKNvtNPXmwdLmNZxxYBDPFkP to 1CoinEXLQtivDjckWMRBbhgVJk9RdwdYbR (5311f9d6d7cca547cd62198094d53a8bb1c837ab721177faaa81ff502ad85e68) [valid]
2015-08-09-T18:55:09+0900 [WARNING] Blockchain reorganisation at block 369063.
2015-08-09-T18:55:09+0900 [INFO] Reparsing all transactions.
2015-08-09-T20:13:45+0900 [INFO] Database minor version number updated.
2015-08-09-T20:32:17+0900 [INFO] Issuance: 1C2iVCgAZFqqVVnGzuea8N8SuWmg6EmGHb created 1000000.0 of divisible asset FANTOKEN (ee1008bf0a14ee62e20c82724bae84e81847b1e34d2b909b6bdc4ddf622a4301) [valid]
2015-08-10-T10:55:49+0900 [WARNING] Blockchain reorganisation at block 369170.
2015-08-10-T10:55:49+0900 [INFO] Reparsing all transactions.
2015-08-10-T12:42:08+0900 [INFO] Database minor version number updated.
2015-08-10-T13:29:25+0900 [INFO] Issuance: 1EhFQ3ycjzXTfSamtNhqHD7fS8LjoBDNa2 created 700000000.0 of divisible asset A2050411063849126000 (daa412cca38f0ca91ce8ee33d9e00bba87b949f724711bef11b71446901f1877) [valid]
2015-08-11-T07:16:42+0900 [WARNING] Blockchain reorganisation at block 369312.
2015-08-11-T07:16:42+0900 [INFO] Reparsing all transactions.
2015-08-11-T11:14:12+0900 [INFO] Database minor version number updated.
2015-08-11-T12:45:07+0900 [INFO] Order: 14wLMb2A9APqrdXJhTQArYLyivmEAf7Y1r ordered 2000.0 SJCX for 25.9996 XCP in 1000 blocks, with a provided fee of 0.0002 BTC and a required fee of 0.0 BTC (f04ade812b4d6f5c8f1ca9374ec4221b2ecca45175e42424d2f06d94c793359e) [open]
2015-08-11-T16:26:44+0900 [WARNING] Blockchain reorganisation at block 369381.
2015-08-11-T16:26:44+0900 [INFO] Reparsing all transactions.
2015-08-11-T20:44:13+0900 [INFO] Database minor version number updated.
2015-08-11-T22:35:25+0900 [INFO] Send: 197.0 SJCX from 1SJCXrYsuWmZzmAhA9K4fYkKqgGyLim79 to 1F6MVcDvqH4QKRR27bfW8Gz1sq9ccJC77V (62a65863b742728248008822840fcb721bc14a4fcd71964a7c9deac4308ef4de) [valid]
When the blockchain reorganizes, the s/w must adjust to it.
That means Bitcoin Core must make changes to the blocks DB, transaction index and address index must be rebuilt.
Additionally, Counterparty must process these changes as well as keep ingesting new transactions.
So it is not possible to avoid this, especially on Bitcoin Core (that can be improved only by bitcoin developers).
Perhaps the only way to improve this is to faster react to reorganizations from the Counterparty side (to faster process changes).
Maybe Bitcoin Core 0.11 behaves better (although I suspect not). Right now there’s no pre-built binary (we’re waiting for BTCDrak to create it), but people can built from source (currently at GitHub - btcdrak/bitcoin at addrindex-0.11). I think v0.12 should be better because there are some performance-related bugs that will hopefully be resolved in v0.12
I think in Counterparty develop there are some smaller improvements related to performance under stress, so probably the next version will behave a bit better, but at the moment I don’t know if it’s possible, and if there’s an effort to make the reorg time much shorter.
It seems your re-orgs take at least 1h30ms or more (even 4 hours?). That’s a lot, but I think probably similar to counterwallet.io (which is why it often seems “down”).
@jdogresorg has a powerful machine with great specs, maybe he can share his reorg performance details.
Reorgs on my servers usually take around 20-30 minutes. Here are the logs from my last few reorgs.
2015-08-09-T08:14:18+0000 [WARNING] Blockchain reorganisation at block 369063.
2015-08-09-T08:14:18+0000 [INFO] Reparsing all transactions.
2015-08-09-T08:35:16+0000 [INFO] Database minor version number updated.
Reorg Time : 20:58
2015-08-10-T01:55:52+0000 [WARNING] Blockchain reorganisation at block 369170.
2015-08-10-T01:55:52+0000 [INFO] Reparsing all transactions.
2015-08-10-T02:16:57+0000 [INFO] Database minor version number updated.
Reorg Time : 21:05
2015-08-10-T22:16:23+0000 [WARNING] Blockchain reorganisation at block 369312.
2015-08-10-T22:16:23+0000 [INFO] Reparsing all transactions.
2015-08-10-T22:45:26+0000 [INFO] Database minor version number updated.
Reorg Time : 29:03
2015-08-13-T06:31:11+0000 [WARNING] Blockchain reorganisation at block 369656.
2015-08-13-T06:31:11+0000 [INFO] Reparsing all transactions.
2015-08-13-T06:57:16+0000 [INFO] Database minor version number updated.
Reorg Time : 26:05
2015-08-13-T11:34:20+0000 [WARNING] Blockchain reorganisation at block 369682.
2015-08-13-T11:34:20+0000 [INFO] Reparsing all transactions.
2015-08-13-T11:56:01+0000 [INFO] Database minor version number updated.
Reorg Time : 21:41
My servers typically have at least 8 cores and 64GB of RAM
My friend has his own counterpartyd server. The server spec is 6 cores and 8GB of RAM. He also said reorganisation takes about 1 hours. It seems that my server spec (4 cores and 8GB of RAM) is not enough.
Anyway, I understood that I cannot avoid it completely. I’ll upgrade my server spec and I’ll prepare the two server: main and sub server. I think that counterwallet.io has such a redundant configuration. I’ll check it out.
As far as I compared the server logs between the jdogresorg’s one and my friend’s one, the reorganisation time are almost same. I don’t think it is effective if I have two counterparty servers and the redundant configuration. What do you think about that?
jdogresorg’s log
2015-08-10-T01:55:52+0000 [WARNING] Blockchain reorganisation at block 369170.
2015-08-10-T01:55:52+0000 [INFO] Reparsing all transactions.
2015-08-10-T02:16:57+0000 [INFO] Database minor version number updated.
2015-08-13-T06:31:11+0000 [WARNING] Blockchain reorganisation at block 369656.
2015-08-13-T06:31:11+0000 [INFO] Reparsing all transactions.
2015-08-13-T06:57:16+0000 [INFO] Database minor version number updated.
My friend’s server. (6 Cores 8GB RAM)
2015-08-10-T01:55:48+0000 [INFO] Blockchain reorganisation at block 369170.
2015-08-10-T01:55:48+0000 [INFO] Reparsing all transactions.
2015-08-10-T02:34:18+0000 [INFO] Checking for conservation of assets.
2015-08-10-T03:23:08+0000 [INFO] Database minor version number updated.
2015-08-13-T06:31:12+0000 [INFO] Blockchain reorganisation at block 369656.
2015-08-13-T06:31:12+0000 [INFO] Reparsing all transactions.
2015-08-13-T07:10:06+0000 [INFO] Checking for conservation of assets.
2015-08-13-T07:53:16+0000 [INFO] Database minor version number updated.
Thanks. I will check the latest version of Bitcoin Core and Counterparty. I will have a new server same as jdogresorg’s spec but still 20-30 minutes down-time is intolerant.