Jump to content
Chan_Maddanna

About time to scale RCL

Recommended Posts

29 minutes ago, Graine said:

It would be nice to see UNL lists include validators from SBI, or other FI (if they are so inclined to participate). Surely, Gatehub and other Ripple gateways should be trust-worthy enough to be included too. 

@Graine I agree generally, but I don't blame them for being over-cautious, i guess.  They are in the lead by a wide margin at the moment (in terms of bank adoption of a possible SWIFT replacement / augmentation). 

Share this post


Link to post
Share on other sites
9 hours ago, RafOlP said:

We had to increase ours and it solved our issues. :)

What did you increase? Fees level or the hardware of the rippled?

Share this post


Link to post
Share on other sites
10 hours ago, JoelKatz said:

Just to be clear, there have been no validator issuers or issues with the core ripple network. The issues have been with the data API, the s1 rippled cluster, and perhaps some servers run by partners. If you were talking to your own instance of rippled, and your server was properly provisioned for the increased load, you would be fine. We've been scaling our services aggressively today and we've heard from partners who are doing so as well.

But you're absolutely right. More scaling is definitely needed all around.

By the way,  if you do notice any instability in your own rippled server, and it's not because the hardware is marginal, the server is under attack, or anything like that, please let support know or let me know by private message. We're keeping a close eye on stability, don't expect any problems but would appreciate any reports.

So is it a matter of hardware?

Would having more rippled around solve some issue, i.e. not sending most of the transactions to ripple's public servers?

Share this post


Link to post
Share on other sites
1 hour ago, tulo said:

So is it a matter of hardware?

Would having more rippled around solve some issue, i.e. not sending most of the transactions to ripple's public servers?

Yes to both, from an external perspective.

It seems the public cluster(s) operated by Ripple Inc. are hit hard, since even some Ripple businesses seem to rely on them instead of operating their own server. IMHO they should be rate limited much more harshly to disincentivize anyone building products expecting high availability of these servers.

However, there are no good guides out there to run your own infrastructure either (like https://github.com/ripple-unmaintained/rippled-ansible-role), so it is hard to tell people to "just install it", if there is not even a useful package out there in any distro repository (the offered RPM wouldn't get accepted anywhere).

Share this post


Link to post
Share on other sites
1 hour ago, Sukrim said:

It seems the public cluster(s) operated by Ripple Inc. are hit hard, since even some Ripple businesses seem to rely on them instead of operating their own server. IMHO they should be rate limited much more harshly to disincentivize anyone building products expecting high availability of these servers.

However, there are no good guides out there to run your own infrastructure either (like https://github.com/ripple-unmaintained/rippled-ansible-role), so it is hard to tell people to "just install it", if there is not even a useful package out there in any distro repository (the offered RPM wouldn't get accepted anywhere).

Yes, I agree to both. Business should run their rippled (also to look more professional) and we need an apt-get install for rippled.

Share this post


Link to post
Share on other sites
3 hours ago, nikb said:

But look at it this way too: despite the connection instability, the RCL kept closing ledgers like clockwork every 3.5 seconds in the face of significantly increased volume - the most important code held up brilliantly and didn't even bat an eyelash; it was rock solid.

Indeed! Kudos to all of you! And to the professional sincere way how you did handle (and now communicate) it.

Share this post


Link to post
Share on other sites

@nikb Thanks for the update!  Grateful for your transparency.

Something that may help reduce load across RCL in general is a "light" or "reduced" book_offers request option.  I suggested this to @JoelKatz a few months ago and he liked the idea.

Typically, 95% of the bandwidth furnished by rippled in response to a book_offers request is thrown away by the client.  Clients who make frequent book_offer requests are usually after two numbers per level of quality: the quality value, and the funded amount of taker_gets.  You can see this in play on exchange API's, where a book request furnishes an ordered array where each member of the array contains only two values for that quality level.  It's all that's required to place a market-accurate order - so it's all they give out.

What do you think of this idea?  David suggested adding the option to show only in detail offers made by the owners account.  A book_offers request could therefore have a couple of new fields.  A boolean "reduced", and an "account" which is optional and only used when reduced is set.  Additionally, a "both" field, to return both sides of the book - consistent with the subscribe version - would be great too.

Getting back to the scaling issue: could you explain what is happening internally within the s1 cluster as a result of this extra load?  Specifically how the consensus process and synchronisation of ledger closes between s1 instances and other instances of rippled has been affected?  From any given s1 instance, I'm regularly seeing book_offers stamped with ledger_indexes up to 100's of ledgers ahead of the ledger_index of the same instances ledger subscription (so - 6 minutes out of sync?).

Share this post


Link to post
Share on other sites
52 minutes ago, Professor Hantzen said:

@nikb Thanks for the update!  Grateful for your transparency.

Something that may help reduce load across RCL in general is a "light" or "reduced" book_offers request option.  I suggested this to @JoelKatz a few months ago and he liked the idea.

Typically, 95% of the bandwidth furnished by rippled in response to a book_offers request is thrown away by the client.  Clients who make frequent book_offer requests are usually after two numbers per level of quality: the quality value, and the funded amount of taker_gets.  You can see this in play on exchange API's, where a book request furnishes an ordered array where each member of the array contains only two values for that quality level.  It's all that's required to place a market-accurate order - so it's all they give out.

What do you think of this idea?  David suggested adding the option to show only in detail offers made by the owners account.  A book_offers request could therefore have a couple of new fields.  A boolean "reduced", and an "account" which is optional and only used when reduced is set.  Additionally, a "both" field, to return both sides of the book - consistent with the subscribe version - would be great too.

Getting back to the scaling issue: could you explain what is happening internally within the s1 cluster as a result of this extra load?  Specifically how the consensus process and synchronisation of ledger closes between s1 instances and other instances of rippled has been affected?  From any given s1 instance, I'm regularly seeing book_offers stamped with ledger_indexes up to 100's of ledgers ahead of the ledger_index of the same instances ledger subscription (so - 6 minutes out of sync?).

It's even better to not use the book_request, but to build the book from the stream of transactions. In that way you have an always updated book in "real-time" and you request the full book only the first time. But that feature is not added into ripple-lib, but it's in a separate library (with problems).

Share this post


Link to post
Share on other sites
38 minutes ago, tulo said:

It's even better to not use the book_request, but to build the book from the stream of transactions. In that way you have an always updated book in "real-time" and you request the full book only the first time. But that feature is not added into ripple-lib, but it's in a separate library (with problems).

Indeed, the subscribe stream is better and more suited.  However, it's more work to build and maintain the book from what you get coming down that stream, and requires a deeper understanding of RCL to implement correctly, which also leaves room for error and the risk of loss from trading against an inaccurate set of offers. Newcomers will go straight for book_offers for these reasons as it's integrity is instantly higher and it's easy to implement.  This may make it a particular pain point during times of an influx of new users.

Share this post


Link to post
Share on other sites
7 hours ago, nikb said:

closing ledgers like clockwork every 3.5 seconds in the face of significantly increased volume -

Thanks for the hard work and detailed answer. Quick question when you get a chance, any reason why RippleCharts is showing that the ledger interval has dropped to 2.87s?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...