Jump to content

the network is down ?


Recommended Posts

16 minutes ago, Soup said:

That sounds great, but no progress was made by the substantiation. It only emboldens FUD, but what do I know??? I'm only HODLing and remaining poor. 

Well, hopefully as a result of this post more people will understand what happened, even if they don’t quite understand all the technical subtleties of the why, how it was addressed and what it means going forward.

The salient point, for me, is that the network needs more active, engaged and passionate participants, that hold each other accountable for the network’s stability and performance as a whole.

Link to post
Share on other sites
  • Replies 68
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

The data that Galgitron is showing would not represent the issue: he's post-processing ledgers based on the close_time field. Here are the facts: sometime around midnight (UTC time!) on 2018-11-1

Nothing casts more doubt that uncertainty and secrets. We all benefit when issues are discussed, analyzed and understood because the net result is a better, stronger network.

Good question. I wish I had a better answer for you, but it boils down to this: The company was using JIRA for everything else, and management wanted to be able to monitor the work we were doing

Posted Images

56 minutes ago, EcneitapLatnem said:

but to be clear, there really would not be a way for an ordinary community member like myself (or the majority of xrp twitter community) to locate and confirm the halt despite being given the date and time it might have happened, correct?

Not really, especially since validations and consensus related messages are ephemeral and not logged or collected anywhere. It also isn't easy to do for someone who might want to start such an archive.

Some indications might be transaction volume going down (since also a lot of nodes were not really working well during that time), but in general just by looking at ledger headers there's no way someone can find out after the fact that for some time some ledgers were only partially validated. Maybe this is different in newer versions of rippled though.

Link to post
Share on other sites
3 minutes ago, Sukrim said:

Not really, especially since validations and consensus related messages are ephemeral and not logged or collected anywhere. It also isn't easy to do for someone who might want to start such an archive.

Some indications might be transaction volume going down (since also a lot of nodes were not really working well during that time), but in general just by looking at ledger headers there's no way someone can find out after the fact that for some time some ledgers were only partially validated. Maybe this is different in newer versions of rippled though.

I think that a project to monitor and archive proposals and validations would be awesome! The newer versions of rippled should make it easier to achieve, because they will, by default, forward all validations as opposed to only trusted validations.

It isn't really a big undertaking if you look at it from a code perspective. But it is much larger if you look at it from the perspective of maintaining the archive, which will grow fast. I would be hugely supportive of anyone choosing to undertake such a project and making the dataset public.

Link to post
Share on other sites

@nikb thanks very much for the detailed information you’ve provided.

Can I ask please, do you think that the commercial entities that are using ODL should all be validating?  Are they already?


 

Btw... I think you might have a typo in an earlier post that reverses your intent:

”I don’t see this as “airing dirty laundry” nor do I feel it’s a good idea for incidents to be discussed and evaluated so lessons can be learned.“

Link to post
Share on other sites
17 minutes ago, BillyOckham said:

@nikb thanks very much for the detailed information you’ve provided.

Can I ask please, do you think that the commercial entities that are using ODL should all be validating?  Are they already?

I think that everyone that relies on the XRP Ledger ought to be an active and engaged participant. That can take many forms, and operating a validator is only one thing they could do.

But let's talk about validators, since you specifically asked about that. Should everyone run a validator? In an ideal world, everyone would be willing to, but I don't think that everyone should. Only those that are able to commit the resources to do it well should, and everyone should be keeping an eye on those that are to ensure that they are doing a good job. I strongly believe that anyone building a business on top of the XRP Ledger should, at the very least, seriously consider investing the time and effort to operate a validator because it is in their best interest to do so and should have a very good reason for choosing not to.

 

17 minutes ago, BillyOckham said:

Btw... I think you might have a typo in an earlier post that reverses your intent:

”I don’t see this as “airing dirty laundry” nor do I feel it’s a good idea for incidents to be discussed and evaluated so lessons can be learned.“

Ooops! You're right!

Link to post
Share on other sites
27 minutes ago, nikb said:

I strongly believe that anyone building a business on top of the XRP Ledger should, at the very least, seriously consider investing the time and effort to operate a validator because it is in their best interest to do so and should have a very good reason for choosing not to.

Thanks for replying with your view.  I agree and have done so for some time.  
 

I believe that many commercial users already do validate,  and to some extent that allays in my mind the concerns that have been raised about network stability.  But I don’t have a firm grasp on the situation and have no metrics to hand about it either.

To be frank though, it bothers me a bit that Ripple have sent such large amounts of funds out to extremely speculative enterprises some of which, so far at least, have done nothing at all with XRP.  I don’t understand why they haven’t spent a tiny percentage of that treasure on hiring or funding an entity to do this enterprise-level network support type work.

I’m fully aware that Ripple don’t own XRPL etc etc, but it’s clearly in their interest and stated aims to support it.  And they already do so in many ways which is something I’m grateful for.  (The work you mentioned earlier to identify, unit test, and bug fix for example.)

But this particular area of network health monitoring seems ready for enterprise level support, and yet doesn’t seem to get it.  If you feel willing and free to comment on this I would much appreciate it.  

Link to post
Share on other sites
3 hours ago, nikb said:

Nothing casts more doubt that uncertainty and secrets. We all benefit when issues are discussed, analyzed and understood because the net result is a better, stronger network.

Very true, but I do not understand why it took you 2 years to come to this conclusion and explaining the incident in detail?  

Link to post
Share on other sites
32 minutes ago, kanaas said:

Very true, but I do not understand why it took you 2 years to come to this conclusion and explaining the incident in detail?  

Fair criticism, and the only thing I can say is mea culpa!

Honestly, I hadn't seen this thread, but that's hardly an excuse. During the incident, I was more focused on figuring out what happened and the immediate next steps needed. Those were days with very little sleep; I think I personally went for almost 5 days, on maybe 30 minutes per day. We kept monitoring the network closely, while working on a post-mortem, and the longer-term fixes necessary. And there also the other million other things I had on my plate, and so. So, the answer is, it just kind of got away from us.

Once it re-surfaced, I came across this thread via Twitter, and couldn't really remember what, if anything, had been published about this; I realized that nobody had written up any details here, so I opted to do it now.

Not ideal and as I said, fair criticism.

Link to post
Share on other sites
8 minutes ago, nikb said:

Once it re-surfaced, I came across this thread via Twitter, and couldn't really remember what, if anything, had been published about this; I realized that nobody had written up any details here, so I opted to do it now.

That’s an excellent reply and I believe we all owe you and the other devs a debt of gratitude.  Feels like sucking up to say that,  but it is the unvarnished truth of the thing I think.

If you are free to, and wish to, I would be grateful if you could look at my last post and comment.  If not,  no worries, you actually owe us all here absolutely nothing.  So thanks for all so far.  :JC_doubleup:

Link to post
Share on other sites
4 hours ago, BillyOckham said:

I believe that many commercial users already do validate,  and to some extent that allays in my mind the concerns that have been raised about network stability.  But I don’t have a firm grasp on the situation and have no metrics to hand about it either.

To be frank though, it bothers me a bit that Ripple have sent such large amounts of funds out to extremely speculative enterprises some of which, so far at least, have done nothing at all with XRP.  I don’t understand why they haven’t spent a tiny percentage of that treasure on hiring or funding an entity to do this enterprise-level network support type work.

I’m fully aware that Ripple don’t own XRPL etc etc, but it’s clearly in their interest and stated aims to support it.  And they already do so in many ways which is something I’m grateful for.  (The work you mentioned earlier to identify, unit test, and bug fix for example.)

But this particular area of network health monitoring seems ready for enterprise level support, and yet doesn’t seem to get it.  If you feel willing and free to comment on this I would much appreciate it.  

While some exchanges operate validators, most aren't (or if they are, they haven't chosen to make that fact public). I don't know of any other commercial users who are running validators; while my permission to do that isn't required and it's possible that they could be without me knowing and opting to remain anonymous, I think it's probably safer to assume that they aren't.

As for improved monitoring, I honestly don't think that the solution is to ask more of Ripple. First, you don't really know what kind of monitoring infrastructure Ripple already has in place, both for its own servers and for the network as a whole. Second, it can't and shouldn't always be Ripple that steps up to do things; if monitoring the network as a whole is important to the community and the ecosystem (and I think it should be) then the community and the ecosystem needs to develop that monitoring capability.

As for your point about funds, my general policy is to not discuss other's holdings or how they choose to deploy them.

Link to post
Share on other sites
39 minutes ago, nikb said:

While some exchanges operate validators, most aren't (or if they are, they haven't chosen to make that fact public). I don't know of any other commercial users who are running validators; while my permission to do that isn't required and it's possible that they could be without me knowing and opting to remain anonymous, I think it's probably safer to assume that they aren't.

As for improved monitoring, I honestly don't think that the solution is to ask more of Ripple. First, you don't really know what kind of monitoring infrastructure Ripple already has in place, both for its own servers and for the network as a whole. Second, it can't and shouldn't always be Ripple that steps up to do things; if monitoring the network as a whole is important to the community and the ecosystem (and I think it should be) then the community and the ecosystem needs to develop that monitoring capability.

As for your point about funds, my general policy is to not discuss other's holdings or how they choose to deploy them.

Thanks Nik for the full and frank reply.  I appreciate your response and your point of view.

I’m pretty sure I read that  @Nicolas from FlashFX said they run a validator.  They use ODL and I may have interpolated that out to infinity.  :)   Maybe an oops??

I do think you are correct that Ripple shouldn’t be the first and only port of call,  but they have such a vested interest and have already done a lot,  so funding this one step further doesn’t seem much of a stretch to me.  
 

I’m also fairly sure (based on knowing their self-interest and capabilities) that Ripple do have monitoring in place and would react if the need arose.  But that is not something that should be relied upon for the greater good of the XRPL so I do hope that the new foundation is considering all this too.

Link to post
Share on other sites
2 minutes ago, BillyOckham said:

Thanks Nik for the full and frank reply.  I appreciate your response and your point of view.

I’m pretty sure I read that  @Nicolas from FlashFX said they run a validator.  They use ODL and I may have interpolated that out to infinity.  :)   Maybe an oops??

You're welcome.

It'd be awesome if FlashFX were running a validator; if they are they ought to make it public.

 

Link to post
Share on other sites
15 minutes ago, nikb said:

You're welcome.

It'd be awesome if FlashFX were running a validator; if they are they ought to make it public.

 

I don’t infest Twitter,  only here,  so I can’t hit him up on Twitter,  but I’m pretty sure he is on there and active.  
 

Perhaps someone reading this will use that wonderful platform, widely known as a wellspring of erudition and civility, to ask him.   :) 

Link to post
Share on other sites

Hi,

The Network need more validator ->probably, at this stage i can't tell.

Do the network need more independent validator run by non professional ->No !

Only participant with capability to professionally maintain a validator according to industry standards of IT infrastructure and compliance (security) should do it. Anyway any serious participant would/ should not add those "low quality" validators to their list.

I can anticipate that any serious participant to the network will run a validator because of their IT risk management. (cost to maintain a validator is almost nothing in an IT budget)

So if the network participants number grow -> the validators "network" will grow accordingly.

I expect with time and adoption (is there is one day) emergence of default validator list "recommended" by a new standard and checked by audit department of the clients companies.

 

To sum-up :

if adoption of new significant participants to the network-> the issue should disappear. 

 

PS : sorry again for my English, hope it make sens

Link to post
Share on other sites
22 hours ago, nikb said:

Nothing casts more doubt that uncertainty and secrets. We all benefit when issues are discussed, analyzed and understood because the net result is a better, stronger network.

So why the JIRA was made private? :rolleyes:

Edited by tulo
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.