Rippled not on current ledger

[cryptodev]

Hi!

I'm running a MainNet Ripple Node (v0.90.1) on an Azure VM. (B4MS, 4 cores, 16gb memory).

Last night it seems it wasn't up with the latest ledger anymore. It kept returning { "error":"noCurrent","error_code":16,"error_message":"Current ledger is unavailable." }. I looked at the logs and noticed it was spamming "Validations:WRN Unable to determine hash of ancestor from ledger hash=[Random hashes here]".

I am running using the node_size set to huge, using nudb and the db is stored on a seperate SSD drive for performance reasons. It is being run as a docker image. The node isn't out of memory afaik, usually doesn't use more than 8gb 

The problem resolved after i restarted the docker container (ie, the rippled process got restarted). +/- 8-9 hours later the problem occurred again.

Any idea what could be wrong or what i could do next time to see what is wrong?

Edited April 25, 2018 by cryptodev

[Sukrim]

Do you have a log analysis and monitoring system configured (e.g. Prometheus for monitoring and ELK stack for logs)? If not, it might be worth gathering some metrics.

 

Alternatively: Restart the container every 6 hours.

[cryptodev]

Not yet.. Perhaps something to look at indeed.

Also, what i found strange is what happens at a restart. Perhaps this is useful and/ or strange, i'm not sure:

Whenever i restart it takes about 20 minutes before it's up to speed with the latest ledger. Then it starts fetching all other ledgers which it already should have in it's local database. I was under the impression that because it has a local db it didn't have to get all the previous ledgers again...?

[cryptodev]

I have set a ledger history of 20000 . It seems each restart means it's fetching all 20k slowly again...

```

[ledger_history]
20000

[node_db]
type=nudb
path=/var/lib/rippled/db/nudb
online_delete=20000
advisory_delete=1
```

 

[Sukrim]

You could start it with "--load" appended instead of "--net", but then startup will take a while because it has to dig through the local database quite a bit. Syncing afterwards is quite fast though because it already has a lot in its cache.