Suggestion: Transaction Fee Tracking

[JoelKatz]

It can sometimes be difficult to choose the right fee to pay for an XRP Ledger transaction. A common approach is to pick a value known to be large enough, such as .0001 XRP, and just using that all the time.

We propose adding code to the server to keep track of when a transaction is first observed and what fee level it chose to pay. Then, when the transaction is seen in a fully-validated ledger, the server can update its statistics for transactions at that fee level. Thus the server could report, at any time, the fraction of successful transactions and the average confirmation time for various fee levels.

In addition to being useful for choosing the fees to pay for a transaction, this would also gather interesting statistical information.

This post is one suggestion for an enhancement to the XRP Ledger. See this post for context:
https://www.xrpchat.com/topic/33070-suggestions-for-xrp-ledger-enhancements/

You can find all the suggestions in one place here: https://coil.com/p/xpring/Ideas-for-the-Future-of-XRP-Ledger/-OZP0FlZQ 

[mDuo13]

I'm all for it. This is also something that could be done on a less-comprehensive level (but perhaps quicker) in frontend/proxy software that handles transaction submission, such as XRP-API.

[Sukrim]

I'm not sure if this really belongs to the back-end of things. These statistics are nice to have but don't have any influence on the actual operation of XRPL, so probably there should be some other software that calculates them. Rippled definitely should be able (or extended) to give the necessary raw data to calculate this stuff, I don't think it should do the calculations.

[DutchBeetle]

As far as I know the transaction fees serve a goal to defend the limitations of the XRPL against spammers. Each ledger chooses the (currently maximum 1500) transactions favoring the transactions which pay the highest fee (high-value to low-value). Therefore a coordinated spam-attack will become very expensive very soon because of the (fast increasing) progressive fees needed to prevent 'normal' transactions a place in the next ledger. And this every 3 to 5 seconds.  

Now, if a 'normal' transaction definitely wants a place in the next ledger, the sender has the possibility to agree a higher fee (for example .0001 XRP as @JoelKatz stated).

I am wondering if there is a possibility that this higher fee is more or less 'optional' and is only used in cases where the ledger really is that full with transactions that it is needed? Wouldn't it be great that the ledger is smart enough to recognize whether this 'higher than minimum fee' is actually needed? And when this fee is excessive an unnecessary the ledger chooses to only 'take out' the necessary minimum? 

In that case, if one definitely wants to make a transaction he/she puts in .0001 XRP as a fee, but when the network doesn't use its full occupancy rate, only .00001 XRP is actually used in this transaction. In the case of a spam-attack, this higher fee is actually needed due to the higher throughput and therefore this higher fee is completely used (or as far is minimally needed for the transaction to take place in this ledger).

An additional advantage will be that these 'fat-finger' faults are very much less harmful. Sometimes you see a transaction which sends .00001 XRP and pays 100XRP in fees, this obviously should have been the other way around. With my proposed alteration this will (normally) result in a transaction of .00001 XRP, costing .00001 XRP also. 

Effectively, the ledger still picks the 1500 highest fee-paying transactions, and downgrades the fees needed for these transactions to 'just' be the top 1500. 

This might be impossible (because I'm no programmer and have very little technical background) but it would be more user-friendly without losing the defense characteristics.

I would love to hear opinions on this, perhaps even from Ripple employees as @JoelKatz  @mDuo13 @warpaul @nikb !

[Sukrim]

If a lower fee than intended is being paid, the additional XRP might end up on someone's account who doesn't want them. Sometimes you are not allowed to send more than necessary and having the system decide dynamically how high fees actually end up being is not a very user friendly thing to do, even if it might save a bit of money over time.

Also you seem to be a bit confused about the number 1500. This is not a part of the protocol (and if you think that this is the "1500 tps" number that came out in load tests a while back: at 3.5 seconds per ledger, this then should be 5250 transactions....).

[DutchBeetle]

1 minute ago, Sukrim said:

If a lower fee than intended is being paid, the additional XRP might end up on someone's account who doesn't want them. Sometimes you are not allowed to send more than necessary and having the system decide dynamically how high fees actually end up being is not a very user friendly thing to do, even if it might save a bit of money over time.

Also you seem to be a bit confused about the number 1500. This is not a part of the protocol (and if you think that this is the "1500 tps" number that came out in load tests a while back: at 3.5 seconds per ledger, this then should be 5250 transactions....).

Automatically deposit the 'overpaid' fees back to the sender,  I can hardly imagine a situation in which this will be conflicting with the actual purpose of the transaction. Seriously, TPS throughput is increased to 5250?.. Wow, nice! (in this case, insert 5250 where is states 1500)

[Sukrim]

No, at 3.5s block time and 1500 tps you reach 5250 transactions in a block. It's simple multiplication. Neither number is in any way realistic and there currently just is no limit in the protocol for transactions submitted + processed in a ledger other than emergent behavior (fees exploding to unreasonable amounts, network/disk limits...). In tests for simplified transactions with most other limits probably removed (high powered machines, GBit connections in the same datacenter) a while back the software itself seems to start to have had issues between 1500 and 1600 transactions a second.The protocol itself could also handle millions or billions or trillions... of transactions a nanosecond, since it is unlimited. There just doesn't seem to be an implementation and hardware out there to actually run it at these rates.

[DutchBeetle]

1 hour ago, Sukrim said:

No, at 3.5s block time and 1500 tps you reach 5250 transactions in a block. It's simple multiplication. Neither number is in any way realistic and there currently just is no limit in the protocol for transactions submitted + processed in a ledger other than emergent behavior (fees exploding to unreasonable amounts, network/disk limits...). In tests for simplified transactions with most other limits probably removed (high powered machines, GBit connections in the same datacenter) a while back the software itself seems to start to have had issues between 1500 and 1600 transactions a second.The protocol itself could also handle millions or billions or trillions... of transactions a nanosecond, since it is unlimited. There just doesn't seem to be an implementation and hardware out there to actually run it at these rates.

Ofcourse, now I understand the 5250:P

[Jerrybo]

What is the goal of that change? What is the current problem that needs to be solved? I'd like to clarify that first for every change. Every change should solve a specific problem.

I like the Idea of Trx fee tracking. From an API consumer side I would like to set a correct fee. Due additional work to get that, I think too often a larger amount is set just "to be sure". With your proposal, it would be easy to get the fee that was used for last 80% succeeded trx for example. Despite that, I think it is still too much effort for many devs/users due the fees are that low. I like the proposal as an easy way to get recent fees, but I cannot see the problem that needs to be solved.

From a practical standpoint, @DutchBeetle 's idea first looks interesting. However, why should the unused amount be returned? I could imaging in reality that would effective double network traffic, due a lot of people choose too large fees. 50% additional "noise" on the network doesn't seem desirable to me. Due the Devs/User decision to not care about current fee, he or she decides in my eyes to ignore the fee costs. I think that’s mostly because they are so tiny. I don't see a problem that needs to be resolved here.

[Sukrim]

1 hour ago, Jerrybo said:

could imaging in reality that would effective double network traffic, due a lot of people choose too large fees.

The unused fee would just not get deducted from the sending account, I don't see how this would increase network traffic...?

[Jerrybo]

Ok, agree. Understood the idea as "send back to sender in separate trx". A deduction before sending wouldn't increase network traffic, thanks for clarification. Anyway, that idea would just solve the problem, that the dev/user is too lazy to care about correct fees. In my eyes it would be basically just an optimization for those wo don't care about fees anyway. If it would be really a problem, participants wouldn't just pick a value known to be large enough.

[DutchBeetle]

24 minutes ago, Jerrybo said:

Ok, agree. Understood the idea as "send back to sender in separate trx". A deduction before sending wouldn't increase network traffic, thanks for clarification. Anyway, that idea would just solve the problem, that the dev/user is too lazy to care about correct fees. In my eyes it would be basically just an optimization for those wo don't care about fees anyway. If it would be really a problem, participants wouldn't just pick a value known to be large enough.

I'd say nowadays it is more or less a wild guess how high the fee must be set for users who definitely want to have their transaction processed ASAP. This alteration would ensure this fast processing but makes 'overpaying' a thing of the past. 

[Guest]

23 hours ago, DutchBeetle said:

Ofcourse, now I understand the 5250:P

Note that transactions don't necessarily mean direct XRP-XRP payments... There are many types of transactions https://xrpl.org/transaction-types.html

Also have a quick peek at how all this is being done (section 7, https://www.scs.stanford.edu/~dm/home/papers/lokhava:stellar-core.pdf) and what @mDuo13 has said in the past https://www.reddit.com/r/Ripple/comments/7zq1uk/ama_with_david_schwartz_on_riama_2272018_at_1230/duz9bvg/?context=3

[JoelKatz]

On 10/2/2019 at 8:41 AM, DutchBeetle said:

I am wondering if there is a possibility that this higher fee is more or less 'optional' and is only used in cases where the ledger really is that full with transactions that it is needed? Wouldn't it be great that the ledger is smart enough to recognize whether this 'higher than minimum fee' is actually needed? And when this fee is excessive an unnecessary the ledger chooses to only 'take out' the necessary minimum? 

I've thought about schemes like this. The problem is that they really only work if everyone participates in fee discovery.

For example, one simple scheme is to include transactions in ledgers with the same rules we currently use but to only charge the lowest of these two rates:

1. The fee level an additional transaction would have had to pay to get included in the ledger.

2. The lowest fee level paid by any transaction in the ledger.

This scheme works very well when the ledger is not "full", since everyone pays the minimum, which is good. And it works well when the ledger is "full" but people really put the maximum they're willing to pay in their transactions and there's a mix of prices. But it fails in two important cases:

1. It doesn't do a good job of accommodating people willing to wait until the next ledger. You either have to choose between being charged a lot to get in the current ledger when you didn't really care about that or potentially not getting in at all.

2. It doesn't work well if everyone just uses the same fee and doesn't think through what they're really willing to pay. For example, imagine if the system is busy and all transactions come from the same wallet software that uses a default maximum fee of 0.001 XRP. Everyone will pay 0.001 XRP. But if the default had been 0.0001 XRP, that would have been what everyone would pay and the same transactions would execute.

So I'm not sure that it would be a significant improvement.

[DutchBeetle]

13 hours ago, JoelKatz said:

This scheme works very well when the ledger is not "full", since everyone pays the minimum, which is good. And it works well when the ledger is "full" but people really put the maximum they're willing to pay in their transactions and there's a mix of prices. But it fails in two important cases:

1. It doesn't do a good job of accommodating people willing to wait until the next ledger. You either have to choose between being charged a lot to get in the current ledger when you didn't really care about that or potentially not getting in at all.

2. It doesn't work well if everyone just uses the same fee and doesn't think through what they're really willing to pay. For example, imagine if the system is busy and all transactions come from the same wallet software that uses a default maximum fee of 0.001 XRP. Everyone will pay 0.001 XRP. But if the default had been 0.0001 XRP, that would have been what everyone would pay and the same transactions would execute.

So I'm not sure that it would be a significant improvement.

My suggestion was that the ledger priority selects the top 1500 fee-paying transactions per second (as it does now if I'm correct) but then only charges the amount which is minimally required to be part of this 'top 1500 per second'. If there are let's say 2000 transaction-requests in this second, the cut off for this ledger will be at the fee willing to be paid by the 500th lowest request. The ledger will then automatically charge the top 1500 out of the 2000 for this amount, but nothing above.. although the requests above this level were willing to pay a higher fee to get in this very ledger. 

1. People who are willing to wait just put in the minimum required and will be processed when there is space in following ledgers

2. The problem with default maximum fees isn't solved, but neither is it nowadays I would argue. It at least lowers the cost of these types of 'mistakes'  and it will definitely lower the cost of absurd priority fees

3. The ledger will still be protected from DoS-type-attacks because overfilling the ledger will still become more and more costly. Being in the top-1500 per second will still be progressively more expensive in this case.