Jump to content
mDuo13

Share your experiences with running rippled

Recommended Posts

1 hour ago, Sukrim said:

Hyper-V (just like any other VM solution) probably just does its job - operating systems are not designed to have their available memory changing at runtime.

Recent operating systems are designed to have their memory changing at runtime. Recent linux versions support this out of the box, Windows also supports this since Windows 7 Vista and Windows Server 2008 2003. Dynamic memory allocation works and this is not main problem.

The main problem is that rippled often consumes all or most of the available memory (despite it is not under any kind of load) and VM resorts to swapping. I'm trying to use dynamic memory just to alleviate this problem.

Quote

If you want to offer access to more system resources, check out Docker (if you're on Linux) or just run the Windows version of rippled.

I'll try compiling Windows version of rippled next week.

1 hour ago, Sukrim said:

Also, if you're only concerned about running a validator on cheap/limited hardware, try to set your nodestore to "none".

My final goal is to have rippled instance that can be used for development and it can be deployed quickly and easily.

 

Edited by T8493

Share this post


Link to post
Share on other sites

I tought, lets give it a try too :)

So, my system is: 

  • Intel Core i5 CPU 750 @ 2.67Ghz, 4 CPU, RAM: 4,00 GB, x64 Windows 10 Professional
  • Samsung SSD 850 EVO 250GB ATA Device

I am running a VM from VMWare:

On this VM I installed CentOS:

I installed rippled exactly as documented here:

Then I ran into some minor issues (which can cause you quite some time to figure out if you run in to  them for the first time). Max file handles on CentOS was set to 1024 and must be increased (google will help you out how to do that on CentOS). Meanwhile I had started rippled with the root account and I think for that reason it no longer started under 'rippled' using the "service rippled start". As I found out in this thread the DB files can be thrown away I threw them away and started all over again. And that seemed to work, rippled started up.

Next thing was changing from RocksDB to NuDB. Both worked and since I have SSD I now stick with NuDB.

Then getting more acqainted with the rippled.cfg (RTFM). In the end I had to change very little. After a few hours trying this and that (and the things described above - of course all at the same time) I came to the following changes in rippled.cfg:

  • [server] port_ws_public
  • [port_ws_public] port = 5006
  • [port_ws_public] ip = <the centOS system IP> 
  • [port_ws_public] protocol = ws
  • [node_size] tiny
  • [node_db] type=nuDB
  • [validators] <my validation pub key>
  • [validation_seed] <my validation secret>

My system now seems to run OK. Next to the 2GB VM I am running another 1GB VM on which I am testing bots etc. I reverted them to the new rippled/validator to see how it behaves and as far as I can tell the bots are working as they are supposed to.

  • CentOS VM - rippled takes +/- 58% of the available 2GB memory. CPU usage is low, +/- 10%, a bit fluctuating
  • My real system memory is at 90% (2GB + 1GB + some more is almost 4GB).
  • SSD usage is fluctuating mostly between the 0% and 50%, sometimes a peek to 100% but definitly not all the time

To conclude,  I am surprised to see rippled running rather well on only a 2GB VM on a not-really-top-of-the-bill-PC (the SSD is good though).

When reading T8493 experiences, I think the main difference seems to be the type of VM. Maybe VMware behaves in this use case better then Hyper-V?

 

 

 

Edited by jn_r

Share this post


Link to post
Share on other sites
3 hours ago, T8493 said:

@jn_r, can you please leave your VM running for one or two days? I'd like to know if your rippled instance also uses all memory.

I was afraid you were gonna ask that ;) I will start it up when I'm back home. It's on the commonly used PC so hopefully the kids will leave it alone..

Share this post


Link to post
Share on other sites
20 hours ago, T8493 said:

@jn_r, can you please leave your VM running for one or two days? I'd like to know if your rippled instance also uses all memory.

Unfortunately my first attempt failed. Not on the SSD or the VM, but on my other (spinning) disc. The spinning disc was constantly at 100% and did not respond anymore. No idea what caused this.

There should be no relation between the spinning disc and the VM and as matter of fact, the VM was still running smoothly after 12 hours. I have restarted my computer now and will give it another try.

My validator btw is "n9MbL5Amp2vpYuLZ4koaUdZV8pNLFXSiU63Fer5Rn81CAUrwSuXm"

Share this post


Link to post
Share on other sites
22 minutes ago, jn_r said:

Unfortunately my first attempt failed. Not on the SSD or the VM, but on my other (spinning) disc. The spinning disc was constantly at 100% and did not respond anymore. No idea what caused this.

I guess your host machine ran out of memory and started swapping because VM consumed majority of (physical) memory.

Share this post


Link to post
Share on other sites
30 minutes ago, T8493 said:

I guess your host machine ran out of memory and started swapping because VM consumed majority of (physical) memory.

I don't know for sure but I think not. My swapping space and OS also run on the SSD and there was no issue with the SSD. It was still running smoothly at 0-10%.

I use the spinning disc only for data storage.  But as it was hanging at 100% it caused the PC not to react normally under the 'main' user. When I logged on via another user all seemed ok and I could check the VM and the validator to be running ok. As I touch the spinning drive then also this user' process freezes. But this user I could log off and then the problem was over - something I could not do with the 'main' user, as I would then also kill the VM. But anyways, long story, my guess still is - not related - let's see how it will behave today.

Share this post


Link to post
Share on other sites
On 21-4-2016 at 11:18 AM, T8493 said:

@jn_r, can you please leave your VM running for one or two days? I'd like to know if your rippled instance also uses all memory.

For the 2nd attempt I have left my rippled validator running for 1 day now. And the results look very good.

I have used the configuration as I described earlier. Looking for causes of the issue I had with my spinning drive the first attempt I reconfigured my VM's CD/DVD (IDE) that was still pointing to the CentOS installation .ISO which was located on that spinning drive. Further no changes made.

My SSD stays well under 10%. At times it wil reach 100% but that usually takes only a few minutes. My guess is that at those times some sort of house keeping is being performed by rippled. Below my typical computer - and specifically the SSD performance:

typical_behavior.jpg

Focussing on the VM, rippled usages ~50% of the available memory.If I interpret correctly, it usages all direct memory (~2Gb) and leaves the swap memory for what it is:

top.jpg

Lastly, the results of my validating attempts. I'm on top just below the ripple.com validators! :smoke:

validator.jpg

 

Very happy with these results of course :) If you want me to do some more measurements or other info from my setup plz let me know. I will will leave the validator running for a few hours but will stop it later today. 

Edited by jn_r

Share this post


Link to post
Share on other sites
3 minutes ago, jn_r said:

Focussing on the VM, rippled usages ~50% of the available memory.If I interpret correctly, it usages all direct memory (~2Gb) and leaves the swap memory for what it is:

No, it uses around 300MB of swap space (285724 KiB).

 

 

Share this post


Link to post
Share on other sites
5 minutes ago, T8493 said:

No, it uses around 300MB of swap space (285724 KiB).

 

 

You're correct, indeed (was looking at the wrong numbers). So +/- 1GB normal Mem used? Wonder how that works, it has 2GB available ... i.e. why use swap if normal mem is available. 

Edited by jn_r

Share this post


Link to post
Share on other sites

Because if memory is very stale, it is smarter to swap it out and use the RAM for more important stuff (like active buffers) than to wait until the last byte is used.

Share this post


Link to post
Share on other sites
2 hours ago, Sukrim said:

Because if memory is very stale, it is smarter to swap it out and use the RAM for more important stuff (like active buffers) than to wait until the last byte is used.

But rippled should release memory which contains stale data.  If rippled can do nothing except validating ledgers with 1GB RAM, then there is no reason why after some time it needs 2GB to perform the same task.

Share this post


Link to post
Share on other sites
21 hours ago, T8493 said:

But rippled should release memory which contains stale data.  If rippled can do nothing except validating ledgers with 1GB RAM, then there is no reason why after some time it needs 2GB to perform the same task.

Why release data it's already loaded and may need unless the system is under memory pressure?

Let rippled and your O/S micro-manage RAM - they can make much better decisions than you could.

Share this post


Link to post
Share on other sites
1 hour ago, nikb said:

Why release data it's already loaded and may need unless the system is under memory pressure?

It seems that rippled (at least in my two cases) doesn't detect the situation when system is under memory pressure and the activity of hard disk is very high because of swapping. And this swapping was caused by rippled.

For example:

  • when VM starts: rippled is just validating and it uses only 1GB => there is no swapping and almost no disk activity, only occasional writes,
  • after 48hrs of uptime: rippled is still just validating and now it uses 2GB => there is heavy disk activity and swapping can use large portion of available disk IOPS.

The VM has 3-4GB of RAM assigned to it.

Edited by T8493

Share this post


Link to post
Share on other sites
1 minute ago, T8493 said:

It seems that rippled (at least in my two cases) doesn't detect the situation when system is under memory pressure and the activity of hard disk is very high because of swapping. And this swapping was caused by rippled.

For example:

  • when VM starts: rippled is just validating and it uses only 1GB => there is no swapping and almost no disk usage, only occasional writes,
  • after 48hrs of uptime: rippled is still just validating and now it uses 2GB => there is heavy disk usage and swapping can use large portion of available disk IOPS.

 

I've never experienced this behavior and I'm running on multiple systems on bother Windows and Linux.

Is this still under Hyper-V or VMware or bare metal?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...