Nodeos Replays

EOSIO is “a highly performant open-source blockchain platform, built to support and operate safe, compliant, and predictable digital infrastructures” and nodeos is “the core service daemon that runs on every EOSIO node”. This article is about the replaying of blockchain blocks using nodeos, which can be necessary to recover from failure or as a faster alternative to synchronising from a p2p network.

To demonstrate and test the concepts involved, I will use my Docker Compose EOSIO services, while referring to the published nodeos documentation. I will step through various test scenarios using a varilink/eosio Docker image based on version 2.0.12 of EOSIO and so any links to the EOSIO documentation within the body of the article will reference the EOSIO v2.0 manual. Since nodeos is the focus of this article, I will not expand on Docker or Docker Compose concepts nor generic command line actions.

Contents

• Start from Genesis
• Successful Shutdown
• Unclean Shutdown
• Replay from Blocks Log
• Create Snapshot
• Replay from Snapshot
• Replay from Snapshot with Blocks Log
• Replay from Snapshot with Partial Blocks Log
• Conclusions

Start from Genesis

I start a node synchronising from genesis with Telos testnet. In order to do this, I need a valid genesis.json file and one or more p2p-peer-address values for Telos testnet.

The ever helpful Telos community publish the genesis.json file and a regularly updated peers.ini file for Telos testnet. However, I prefer to generate a fresh list of p2p-peer-address values using the “Get Peers” tool in my Telos toolbox.

I put the genesis.json file and peers.ini (renamed to config.ini) files into the ./config folder of my Docker Compose project and run the nodeos service:

docker-compose run --rm nodeos --delete-all-blocks --genesis-json /config/genesis.json

The –delete-all-blocks option is only needed to clear down whatever might be persisted from previous runs in the eosio-data Docker volume.

Within a couple of minutes, the node has started to request and receive blocks from the p2p network. I use the Docker Compose data service:

docker-compose run --rm data

and examine the contents of the /data directory:

/data
|-- blocks
|   |-- blocks.index
|   |-- blocks.log
|   `-- reversible
|       `-- shared_memory.bin
|-- snapshots
`-- state
    |-- fork_db.dat
    `-- shared_memory.bin

I will ignore the blocks/reversible and snapshots directories for now.

As the synchronisation of my node is in progress, it is pulling blocks from the p2p network in their correct sequence from genesis and appending them to the blocks.log file. The blocks.index file serves, as the name would suggest, as an index for querying the blocks.log file for specific blocks.

I can see that both blocks.log and blocks.index are increasing in size as a result of the ongoing synchronisation and that blocks.index is significantly smaller in size than blocks.log. All of this is as I would expect from an elementary understanding of their nature.

The content of the state directory is the chain state or chain database. The blocks contain transactions. As transactions are executed, they may update data held in tables that are associated with accounts on the chain. The chain database contains those tables and its state reflects all updates from genesis to the last block for which transactions have been applied.

Further elaboration of these concepts can be found under Storage and Read Modes in the EOSIO manual.

Successful Shutdown

Since by default Docker Compose runs attached to the console, I can now shutdown my running node using Ctrl+C. The successful shutdown is confirmed at the end of the log:

nodeos successfully exiting

and then I can simply bring the service back up again as no special command line options are needed now:

docker-compose up nodeos

The node resumes its synchronisation activity. This time it was not necessary to set the –genesis-json option because we had a valid chain database in place. So, blocks can be retrieved and the transactions within those blocks applied to the chain database to continue to move it on in time.

Unclean Shutdown

Using this containerised setup, I can now very easily demonstrate the consequences of an unclean shutdown of the node such as happens for example if there is a power failure. I simply kill the running Docker container:

docker kill eosio-nodeos

This time I do not see the nodeos successfully exiting message at the bottom of the log, instead I see:

eosio-nodeos exited with code 137

If I now try to restart the node again:

docker-compose up nodeos

the service fails to come up, with the following message at the end of the log:

database dirty flag set (likely due to unclean shutdown): replay required

This is a failure scenario that we have experienced on occasion at Telos UK, triggered by our hosting company experiencing a fault that power cycled our nodeos servers. A normal restart is not possible because the unclean shutdown of nodeos.

I wouldn’t want to go back to genesis, and throw away all my synchronisation activity to date, so I need another approach to return to synchronising without throwing away my synchronisation to date. The resolution path is shown under Nodeos Troubleshooting in the EOSIO manual.

Replay from Blocks Log

The topic Nodeos Replays is well documented in the EOSIO manual. The documentation states “Replaying data can be done in two ways, from a blocks.log file… from a snapshot file”. I have yet to create a snapshot file in my tests so I’ll recover from blocks.log.

How to replay from a blocks.log file in the EOSIO manual, instructs me to remove files from my nodeos data directory. Again, I can gain shell access to my nodeos data directory using my Docker Compose data service:

docker-compose run --rm data

I remove all the files that are present for which “Remove” is indicated as the action to take in How to replay from a blocks.log file. I already have the blocks.log file in place that has been generated from my synchronisation activity to date and no alternative blocks.log file to replace it with, so I will stick with the blocks.log file that I already have for now.

Now I can replay:

docker-compose run --rm nodeos --replay-blockchain

I can see that nodeos starts to replay from block 2:

existing block log, attempting to replay from 2 to 62443 blocks

I think that this is because block 1 in the blocks.log contains the genesis state, hence why I did not have to refer to my genesis.json file? Of course this also tells me that my node had managed to synchronise with the Telos testnet up to block 62,443 prior to the unclean shutdown.

The replay of blocks is slow initially but soon accelerates. Perhaps this is because the density of transactions in the early blocks for Telos testnet is much higher? If I use my Docker Compose project’s data service again to examine the contents of my nodeos data directory, I can see that blocks.index and the chain database are being updated but not yet blocks log.

Eventually, the replay from blocks.log is exhausted, p2p network synchronisation resumes and blocks.log starts growing again.

Again, all of this is as I would expect based on an elementary understanding of what’s happening here.

Create Snapshot

Now I will demonstrate/test the concepts described in How to replay from a snapshot in the EOSIO manual. Of course, the first step is to create a snapshot to use.

The EOSIO manual documents How to generate a snapshot but my Docker Compose EOSIO services contains a convenience “snapshot” service to do this. Again, I stop the running nodeos service gracefully using Ctrl+C and run my snapshot service:

docker-compose run --rm snapshot

The service has a pause in it to allow nodeos to start up but in a short while a snapshot is created and reported on and the container exits.

{
  "snapshot_name" : "/data/snapshots/snapshot-0002a26212a61554e03b83d2cc47b14247f1d486dbfe3a638a5f3c08cf1a1d3a.bin",
  "head_block_id" : "0002a26212a61554e03b83d2cc47b14247f1d486dbfe3a638a5f3c08cf1a1d3a"
}

Note that the head_block_id corresponding to the snapshot point is also reflected in the snapshot_name. Of course it corresponds to the synchronisation point that my test node had reached when I created the snapshot.

I bring the nodeos service back up again and leave it a while to synchronise with the p2p network beyond the block corresponding to the snapshot. Then, since I’m interested in recovery from failure scenarios, I once again kill the docker container for my running nodeos Docker Compose service to create an unclean shutdown.

Replay from Snapshot

In How to replay from a snapshot in the EOSIO manual it says:

When replaying from a snapshot file it is recommended that all existing data is removed, however if a blocks.log file is provided it must at least contain blocks up to the snapshotted block and may contain additional blocks that will be applied as part of startup

Consistent with the recommendation, I will begin by replaying from the snapshot file I have just created with no blocks.log file in situ. Before I do this however, I use my Docker Compose data service to gain shell access to my nodeos data directory and take a backup copy of blocks.log to a new directory, /data/backup (it’s important that this directory is within my data Docker volume so that it is persisted across container runs).

Then, following the instructions in How to replay from a snapshot in the EOSIO manual, I remove the /data/blocks and /data/state directories, leaving only my snapshot located in /data/snapshots in place.

The I start the replay from the snapshot:

docker-compose run --rm nodeos --snapshot /data/snapshots/snapshot-0002a26212a61554e03b83d2cc47b14247f1d486dbfe3a638a5f3c08cf1a1d3a.bin

Of course, this command refers to the specific snapshot file that I generated earlier. Note that the documentation suggests that using a combination of –snapshots-dir to indicate the directory containing the snapshot file and a value for –snapshot that is the snapshot filename within that directory will work. I’ve never been able to make that work. Every time nodeos reports that the snapshot file doesn’t exist for me if I try that approach.

After a the initial start-up messages have passed, the log pauses for a little while at the message “Starting initialization from snapshot, this may take a significant amount of time” and then the node resumes normal synchronisation with the p2p network. I then shutdown the nodeos service normally and then bring it back up again. Synchronisation with the p2p network continues.

If I use the cleos service in my Docker Compose EOSIO services to get the block corresponding to the point that the snapshot was generated:

docker-compose run --rm cleos get block 0002a26212a61554e03b83d2cc47b14247f1d486dbfe3a638a5f3c08cf1a1d3a

I receive the following response:

Could not find block: 0002a26212a61554e03b83d2cc47b14247f1d486dbfe3a638a5f3c08cf1a1d3a

This provides an opportunity to demonstrate the eosio-blocklog tool that comes with EOSIO. I find this tool to be very useful. I open a shell session within my nodeos data directory using my Docker Compose data service and use this tool to smoke test my blocks directory:

eosio-blocklog --smoke-test

This returns:

Smoke test of blocks.log and blocks.index in directory "/data/blocks"
block log version= 3
first block= 172643
last block= 215940
blocks.log and blocks.index agree on number of blocks

Note that the “first block” reported is 172,643, which is of course well beyond the genesis point. It is also the first block after the snapshot point, which I confirm in my next test.

Replay from Snapshot with Blocks Log

I shutdown the nodeos service normally again using Ctrl+C. In my open shell session within my nodeos data directory, I remove the contents of the /data/blocks and the /data/state directory entirely. I copy the blocks.log copy that I made earlier, from /data/backup to /data/blocks directory and run this command:

eosio-blocklog --make-index

This returns:

Will read existing blocks.log file /data/blocks/blocks.log
Will write new blocks.index file /data/blocks/blocks.index
block log version= 3
first block= 1         last block= 323957
eosio-blocklog - making index took 64 msec

So, as long as I have a valid blocks.log, I can always use eosio-blocklog to rebuild the blocks.index. Note that the “last block” in my restored blocks.log is well beyond the snapshot point as a result of my letting synchronisation run on beyond when I took the snapshot earlier.

Now I replay again from the snapshot as I did earlier:

docker-compose run --rm nodeos --snapshot /data/snapshots/snapshot-0002a26212a61554e03b83d2cc47b14247f1d486dbfe3a638a5f3c08cf1a1d3a.bin

This time, after the message “Starting initialization from snapshot, this may take a significant amount of time” the node goes into a replay from the blocks.log and only when it has exhausted that does it then resume synchronisation with the p2p network.

I shutdown the nodeos service normally and bring it up again and once again query the block corresponding to the snapshot:

docker-compose run --rm cleos get block 0002a26212a61554e03b83d2cc47b14247f1d486dbfe3a638a5f3c08cf1a1d3a

This time I get a successful response that includes:

{
  …
  "block_num": 172642,
  …
}

This confirms that my snapshot was at that block number as I asserted earlier. Now I query the chain state that my node is currently synchronised to:

docker-compose run --rm cleos -u get info

The output includes:

{
  …
  "head_block_num": 341555,
  …
}

I can clearly see the result of the replay to the end of the block.log that I restored followed by a period of p2p network synchronisation.

Replay from Snapshot with Partial Blocks Log

Now I will dig into the statement in How to replay from a snapshot in the EOSIO manual:

“if a blocks.log file is provided it must at least contain blocks up to the snapshotted block and may contain additional blocks that will be applied as part of startup”

When I first read this, I thought I knew the implications of what it is saying but I wasn’t sure. So, I resolved to run some tests to confirm the exact behaviour. I repeated the Replay from Snapshot with Blocks Log test but after having restored my blocks.log file, I used the –trim-blocklog option of eosio-blocklog with various combinations of –first and –last options in order to trim blocks.log prior to building the blocks.index and then replaying from the snapshot each time.

Here are the scenarios tested in this way and the observed results each time:

Scenario	First	Last	Result
blocks.log falls short of snapshot block	N/A	172641	Replay fails with message: Block log is provided with snapshot but does not contain the head block from the snapshot nor a block right after it
blocks.log up to snapshot block	N/A	172642	Replay succeeds – nodeos immediately goes into synchronisation with the p2p network with no replay from the blocks.log.
blocks.log does not contain genesis but spans the snapshot block	150000	200000	Replay succeeds – nodeos initially replays from the blocks.log until that is exhausted and then goes into synchronisation with the p2p network.
blocks.log from the block after the snapshot block.	172643	N/A	Replay succeeds – nodeos initially replays from blocks.log until that is exhausted (which of course is later than in the previous test) and then goes into synchronisation with the p2p network.
blocks.log starts after the block following the snapshot block	200000	N/A	Replay fails with message: Block log is provided with snapshot but does not contain the head block from the snapshot nor a block right after it

Overall, we can clearly see that following any successful replay from a snapshot our blocks.log will:

1. Contain all the blocks from no later than the first block after the snapshot to the current synchronisation point, with no gaps
2. Could contain blocks from earlier than the first block after the snapshot, again with no gaps between the first block it contains and the current synchronisation point
3. Does not necessarily contain all the blocks from genesis

Conclusions

1. A snapshot is a vital enabler of swift recovery from the unclean shutdown of nodeos. It is important to have a strategy in place to regularly create them, so that you know you always have a recent one to hand, relative to the current chain head block.
2. Replay using a snapshot with a blocks.log in situ brings two further benefits:
   1. It can facilitate replay from a snapshot while retaining blocks in blocks.log that precede the snapshot.
   2. If it contains blocks after the snapshot then that enables a faster return to synchronisation with the current chain state than synchronisation with the p2p network alone.
3. In How to replay from a snapshot in the EOSIO documentation it states, “When replaying from a snapshot file it is recommended that all existing data is removed, however if a blocks.log file is provided… ”. This implies to me that there is something suboptimal about replaying from a snapshot with a blocks.log file provided. Based on my investigations I cannot discern why that might be and would welcome comments on that point from anybody with greater insight.
4. Similarly, in Nodeos Replays in the EOSIO documentation it describes two options for replaying from the blocks.log, –replay-blockchain and –hard-replay-blockchain. It says about –hard-replay-blockchain that “This option assumes that the backup blocks.log file may contain corrupted blocks.” I observe that –replay-blockchain has been fine for all the tests described in this article and so I am curious as to what circumstances would require –hard-replay-blockchain to be used. Again, I would welcome comments on that point from anybody with greater insight.