Jump to content
mDuo13

Share your experiences with running rippled

Recommended Posts

1 hour ago, nikb said:

As for the 1GB cache: the O/S will aggressively use RAM as a cache. This makes sense: you don't want RAM to be sitting empty, it does you no good. So if you have, say, 4GB of RAM but the programs you run only need 1.2GB, then O/S will do its best to use the remaining 2.8GB. And what can it use it as other than cache? If the guest came under memory pressure, it would reduce that 1GB rapidly by just throwing away large chunks of the cache.

What is more optimal for rippled in the case when 2GB of RAM is assigned to VM:

  • 1GB of RAM is used for rippled and other processes + 1GB is used for disk cache OR
  • 2GB of RAM is used by rippled and other processes + 0GB is used for disk cache? (assuming there is no memory swapping)
Quote

Generally, the amount of physical RAM available on a machine is fixed for the duration of execution. There are some exotic architectures where this isn't the case, but those require operating system support and try to make this transparent to actual applications.

In virtual environments it is actually quite common for hypervisor to reclaim memory from the virtual machines. CentOS as a guest OS should support this. This is roughly equivalent to pulling one RAM module from the physical machine while the machine is still running.

Share this post


Link to post
Share on other sites
1 hour ago, T8493 said:

What is more optimal for rippled in the case when 2GB of RAM is assigned to VM:

  • 1GB of RAM is used for rippled and other processes + 1GB is used for disk cache OR
  • 2GB of RAM is used by rippled and other processes + 0GB is used for disk cache? (assuming there is no memory swapping)

I'll try my best to give you an answer although I think your question is slightly flawed. First a bit of explanation: The O/S uses some amount of RAM for caching, and then, on top of that, also uses "free" memory (i.e. memory that processes don't actively need) for that purpose as well. The idea is to not waste a resource when it could be put to good use. It's hard to answer the question - it depends on what one does with the machine. Most modern operating systems are really good about giving processes the resources that they need to get their job done. In other words, if the system is telling you that processes are using up 1GB of RAM and the cache is using up another, then that should be an indication that, combined and at query time, the processes have only really asked for about 1GB of physical memory. It's likely that later they'll want more, and when that happens the cache will shrink. Now with all that said: rippled likes memory, and with only 2GB and a bunch of other processes running (including, presumably, X-Windows, etc) you're imposing resource restrictions on the operating system, especially for a nominal server machine.

1 hour ago, T8493 said:

In virtual environments it is actually quite common for hypervisor to reclaim memory from the virtual machines. CentOS as a guest OS should support this. This is roughly equivalent to pulling one RAM module from the physical machine while the machine is still running.

Typically, the hypervisor knows how much RAM the machine is actively using and will "steal" some of it back if the host or another VM is under memory pressure. But this kind of RAM stealing shouldn't be common - if it is, it indicates that either the guest is overloaded with VMs or is underprovisioned with resources.

Now, as to your other comment: most systems don't really support hot-pluggable RAM. Some high-end server machines do, and operating systems geared towards such servers  (like CentOS) do have support for it, but it's by no means automatic and requires administrator intervention. This about what it means for a RAM (that's likely populated with data) to be randomly unplugged. What happens when a program goes looking for the data that was on that RAM stick? It will almost certainly go: "well, dang... it was right here when I left!" and then it crashes, because its precious data is gone...

Under some paravirtualized systems,  it's possible for the host and the guest to cooperate, allowing the host to inform the guest that it's under memory pressure and that it needs to reduce its memory footprint, allowing the host to "steal" some of the RAM back. And yes, if the guest has the sort of advanced "hot-pluggable" RAM support it can maybe leverage it to make the transition unnoticeable to programs by adjusting virtual page tables accordingly. But general purpose server programs aren't likely to know or care about such a situation - that's for the O/S to abstract away.Typically, however, what will happen is that the host will actually swap the guest's RAM and the guest will continue to run - a little slower and with more disk crunching - but generally oblivious to the memory pressure

 

Share this post


Link to post
Share on other sites
1 hour ago, nikb said:

Typically, the hypervisor knows how much RAM the machine is actively using and will "steal" some of it back if the host or another VM is under memory pressure. But this kind of RAM stealing shouldn't be common - if it is, it indicates that either the guest is overloaded with VMs or is underprovisioned with resources.

Hmmm, not necessarily. In my setup host machine was clearly not overloaded with VMs and I didn't use it for anything else than just browsing. Hyper V actually "steals" RAM all the time - it starts virtual machine with "Startup RAM" and after some time assigns another amount of memory to this VM that is between "Minimum RAM" and "Maximum RAM". You can disable this behaviour, but then you must provide fixed amount of RAM for this VM. However, this should be around 4GB, which is not practical unless you have 12 or 16GB RAM in your host machine. Just Visual Studio and some necessary tools (MSSQL Server, IIS Express, Linqpad, Mathematica) consume 5+ GB of RAM.

However, what I experience is that rippled/VM behaves very differently when:

  • guest machine runs with fixed amount of RAM (2GB),
  • guest machine starts with some amount of RAM, but then hypervisor reduces assigned memory to 2GB.

In both cases there is the same amount of memory assigned to VM, but in the second case performance can be much worse sometimes. Although the second case is actually preferred because the guest VM can request additional RAM if it needs to. In the first case guest VM must resort to swapping very quickly.

I understand that rippled doesn't expect that the "physical" RAM shrinks dramatically. But there should be an option for developers to run rippled in VM on their development machines. Right now this seems slightly iffy if developer doesn't have at least 12-16GB RAM in her machine. But many dev machines, especially laptops, have only 6 or 8 GB RAM.

I'm also not sure how developer can run rippled on a non-dedicated server because rippled (+basic daemons) seems to always use around 50% of available RAM (approximately; tested with 2, 3, 4, 5 GB RAM; this is amount of RAM that rippled sees when the machine starts in case when the RAM is dynamically allocated) and even setting node_size to tiny doesn't seem to help. However, I haven't tried all this in practice because all of my servers run ubuntu.

I think option like node_size tiny should limit amount of RAM that the rippled instance uses.

So node_size tiny or something equivalent would imply that amount of RAM used by rippled will always be <= 1 GB or something like that.

 

 

 

Edited by T8493

Share this post


Link to post
Share on other sites
1 hour ago, T8493 said:

Hmmm, not necessarily. In my setup host machine was clearly not overloaded with VMs and I didn't use it for anything else than just browsing. Hyper V actually "steals" RAM all the time - it starts virtual machine with "Startup RAM" and after some time assigns another amount of memory to this VM that is between "Minimum RAM" and "Maximum RAM". You can disable this behaviour, but then you must provide fixed amount of RAM for this VM. However, this should be around 4GB, which is not practical unless you have 12 or 16GB RAM in your host machine. Just Visual Studio and some necessary tools (MSSQL Server, IIS Express, Linqpad, Mathematica) consume 5+ GB of RAM.

You may not have overloaded your machine with VMs and limited yourself to only browsing, yet clearly the machine felt that it was under enough memory pressure to reduce the amount of RAM available to the virtual machine, causing it memory pressure. I have no insight on how Hyper-V makes decisions about when to reclaim RAM; perhaps it is tuned to provide good interactive response to the host, or maybe it sees that the VM isn't using all the available RAM, so it trims back a bit. Maybe it's something entirely different, but either way, general purpose programs can't be expected to adjust to that sort of thing.

 

2 hours ago, T8493 said:

However, what I experience is that rippled/VM behaves very differently when:

  • guest machine runs with fixed amount of RAM (2GB),
  • guest machine starts with some amount of RAM, but then hypervisor reduces assigned memory to 2GB.

In both cases there is the same amount of memory assigned to VM, but in the second case performance can be much worse sometimes. Although the second case is actually preferred because the guest VM can request additional RAM if it needs to. In the first case guest VM must resort to swapping very quickly.

That's not unexpected. If you went and bought a Ferrari F12berlinetta, you would be surprised too when, a couple of weeks later, while you are trying to pass a car on the highway, the engine doesn't pull because Ferrari realized you normally don't drive full-out and replaced the engine with one from a Fiat 500 along with a promise to put the original engine back in if you ever need it.

 

2 hours ago, T8493 said:

I understand that rippled doesn't expect that the "physical" RAM shrinks dramatically. But there should be an option for developers to run rippled in VM on their development machines. Right now this seems slightly iffy if developer doesn't have at least 12-16GB RAM in her machine. But many dev machines, especially laptops, have only 6 or 8 GB RAM.

I routinely run rippled on a laptop with 6GB of RAM and I don't really have any issues. Sure, I'm not running it inside a VM but on the bare metal instead, because I know that the machine just simply isn't beefy enough to run a VM.

 

2 hours ago, T8493 said:

I'm also not sure how developer can run rippled on a non-dedicated server because rippled (+basic daemons) seems to always use around 50% of available RAM (approximately; tested with 2, 3, 4, 5 GB RAM; this is amount of RAM that rippled sees when the machine starts in case when the RAM is dynamically allocated) and even setting node_size to tiny doesn't seem to help. However, I haven't tried all this in practice because all of my servers run ubuntu.

I think option like node_size tiny should limit amount of RAM that the rippled instance uses.

So node_size tiny or something equivalent would imply that amount of RAM used by rippled will always be <= 1 GB or something like that.

I am curious - why not compile and run rippled directly on the Windows bare metal?

Share this post


Link to post
Share on other sites
2 hours ago, nikb said:

I am curious - why not compile and run rippled directly on the Windows bare metal?

I completely forgot about this option. But I have a feeling it will be a 8+ hrs journey again ;) I'll try this next week if I can't manage to buy 16 GB RAM till then. 8GB unbuffered non ECC modules are not very common.

 

Edited by T8493

Share this post


Link to post
Share on other sites

I tried to get it to run on a RaspberryPi by the way once (though the issue was mostly with ARM architecture and missing libraries or at least compilation problems) and have definitely run rippled semi successfully on a 2 GB RAM Intel ATOM Nettop box. If you need 16 GB or RAM for it, there might be something else going on... My "huge" node uses 7.1 GB RAM right now, so I don't see any problem on running it with 8 GB. The 4 GB minspec is mostly for compilation (the code is structured in a way to load as much as possible in RAM and then compile instead of compiling tiny files and then loading them from disk again) as far as I understood. I'll have to try out the small machine though, it was a while since I've booted that one.

Share this post


Link to post
Share on other sites
14 minutes ago, Sukrim said:

I tried to get it to run on a RaspberryPi by the way once (though the issue was mostly with ARM architecture and missing libraries or at least compilation problems) and have definitely run rippled semi successfully on a 2 GB RAM Intel ATOM Nettop box. If you need 16 GB or RAM for it, there might be something else going on... My "huge" node uses 7.1 GB RAM right now, so I don't see any problem on running it with 8 GB. The 4 GB minspec is mostly for compilation (the code is structured in a way to load as much as possible in RAM and then compile instead of compiling tiny files and then loading them from disk again) as far as I understood. I'll have to try out the small machine though, it was a while since I've booted that one.

I'm trying to run it in a virtual machine. I need 12-16GB if I want to do anything else than browsing on the host machine. If I assign 4GB RAM to my virtual machine (and I'm trying to avoid this), then I'm left with only 4GB on my system which is not enough for my work. And I'm doing just typical developer tasks - compilation, running tests, SQL query optimizations, etc.

Edited by T8493

Share this post


Link to post
Share on other sites

I'm testing rippled in a VM again. When the machine is restarted and rippled synchronizes with the rest of the network, VM consumes around 1GB RAM (measured with "free", column "used"). However, after this rippled starts consuming more and more memory until VM consumes almost all memory assigned to it (2GB). Memory usage rises at the rate about 100MB per hour of uptime.

This rippled instance is just validating and it is not under any kind of external load.

Is there a way to prevent this?

Edited by T8493

Share this post


Link to post
Share on other sites
1 hour ago, T8493 said:

I'm testing rippled in a VM again. When the machine is restarted and rippled synchronizes with the rest of the network, VM consumes around 1GB RAM (measured with "free", column "used"). However, after this rippled starts consuming more and more memory until VM consumes almost all memory assigned to it (2GB). Memory usage rises at the rate about 100MB per hour of uptime.

This rippled instance is just validating and it is not under any kind of external load.

Is there a way to prevent this?

What version of rippled are you running? Can you share your config (if you do so then DON'T INCLUDE YOUR [validation_seed]!) so I can try and replicate your configuration?

Share this post


Link to post
Share on other sites
2 hours ago, nikb said:

What version of rippled are you running? Can you share your config (if you do so then DON'T INCLUDE YOUR [validation_seed]!) so I can try and replicate your configuration?

0.30.1-hf2

Right now after 2 hours of uptime memory usage is around 1.1GB and it was around 0.86GB after I restarted the VM.

However, memory usage was around 1.3GB several minutes ago. This high memory usage coincided with extreme disk utilization - dstat in VM showed around 100-200MB/s reads for several consecutive seconds and windows task manager was showing almost 100% disk utilization for several minutes.

Currently, free shows that around 370MB swap space is used and dstat shows there are almost no disk reads (1-5MB read every few seconds).

Host machine has enough free RAM all the time.

EDIT: Btw, I don't think it is a memory leak. It is just that rippled sometimes behaves like it was the only process that needs computer resources....

 

Quote

 

[server]
port_rpc_admin_local
port_peer
port_ws_admin_local
#port_ws_public
#ssl_key = /etc/ssl/private/server.key
#ssl_cert = /etc/ssl/certs/server.crt

[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http

[port_peer]
port = 51235
ip = 0.0.0.0
protocol = peer

[port_ws_admin_local]
port = 6006
ip = 127.0.0.1
admin = 127.0.0.1
protocol = ws

#[port_ws_public]
#port = 5005
#ip = 127.0.0.1
#protocol = wss

#-------------------------------------------------------------------------------

[node_size]
# medium
#tiny
tiny

# This is primary persistent datastore for rippled.  This includes transaction
# metadata, account states, and ledger headers.  Helpful information can be
# found here: https://ripple.com/wiki/NodeBackEnd
# delete old ledgers while maintaining at least 2000. Do not require an
# external administrative command to initiate deletion.
#[node_db]
#type=RocksDB
#path=/var/lib/rippled/db/rocksdb
#open_files=2000
#filter_bits=12
#cache_mb=256
#file_size_mb=8
#file_size_mult=2
#online_delete=2000
#advisory_delete=0

[node_db]
type=nudb
path=/var/lib/rippled/db/nudb
online_delete=2000
advisory_delete=0

[database_path]
/var/lib/rippled/db

# This needs to be an absolute directory reference, not a relative one.
# Modify this value as required.
[debug_logfile]
/var/log/rippled/debug.log

[sntp_servers]
time.windows.com
time.apple.com
time.nist.gov
pool.ntp.org

# Where to find some other servers speaking the Ripple protocol.
#
[ips]
r.ripple.com 51235

# Public keys of the validators that this rippled instance trusts.  The latest
# list of validators can be obtained from https://ripple.com/ripple.txt
#
# See also https://wiki.ripple.com/Ripple.txt
#
[validators]
n949f75evCHwgyP4fPVgaHqNHxUVN15PsJEZ3B3HnXPcPjcZAoy7    RL1
n9MD5h24qrQqiyBC8aeqqCWvpiBiYQ3jxSr91uiDvmrkyHRdYLUj    RL2
n9L81uNCaPgtUJfaHh89gmdvXKAmSt5Gdsw2g1iPWaPkAHW5Nm4C    RL3
n9KiYM9CgngLvtRCQHZwgC2gjpdaZcCcbt3VboxiNFcKuwFVujzS    RL4
n9LdgEtkmGB9E2h3K4Vp7iGUaKuq23Zr32ehxiU8FWY7xoxbWTSA    RL5

# The number of validators rippled needs to accept a consensus.
# Don't change this unless you know what you're doing.
[validation_quorum]
3

# Turn down default logging to save disk space in the long run.
# Valid values here are trace, debug, info, warning, error, and fatal
[rpc_startup]
{ "command": "log_level", "severity": "warning" }

# If ssl_verify is 1, certificates will be validated.
# To allow the use of self-signed certificates for development or internal use,
# set to ssl_verify to 0.
[ssl_verify]
1


[validation_seed]
ssss

 

 

Edited by T8493

Share this post


Link to post
Share on other sites
34 minutes ago, Sukrim said:

The heavy disk activity is probably online delete kicking in. You set it quite low, so it will come into effect every ~2 hours or so.

Is it possible to reliably confirm this (using the log file or something)?

Share this post


Link to post
Share on other sites
6 hours ago, T8493 said:

Is it possible to reliably confirm this (using the log file or something)?

You should be able to see log entries beginning with "rotating  validatedSeq"

Share this post


Link to post
Share on other sites

I must report rippled consumed almost all available memory and swap space on my VM and it had to be restarted. This VM has 2GB of RAM assigned to it and it is configured with 1.2GB swap space. When I restarted this VM and after rippled synchronized with the network, this VM used around 0.85GB of RAM and 0 swap space. But now after almost 2 days of uptime, it uses 1.2GB of RAM and 0.9GB of swap space and disk works like crazy. Rippled uses majority of swap space.

I don't know why Hyper-V doesn't increase the amount of RAM assigned to this VM. Host machine has enough free RAM. The amount of assigned RAM was always the same (2GB).

This rippled instance is just validating and it is not under any kind of external load.

 

 

 

Edited by T8493

Share this post


Link to post
Share on other sites

Hyper-V (just like any other VM solution) probably just does its job - operating systems are not designed to have their available memory changing at runtime. If you want to offer access to more system resources, check out Docker (if you're on Linux) or just run the Windows version of rippled.

Also, if you're only concerned about running a validator on cheap/limited hardware, try to set your nodestore to "none".

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...