Jump to content

Front-running (?) and other questions


Eik

Recommended Posts

I have been out of following Ripple etc for quite some time (before ILP), but recently made some more time for this again. So excuse my missing out on things. I hope someone can help me out with this front-running thing which has been on my mind I guess ever since I started with Ripple, way before http://availableimagination.com/exploiting-ripple-transaction-ordering-for-fun-and-profit/ (

I am on this topic again, not just because someone can make some profit out of my flexibility of what I would like to pay as maximum for a trade, but mainly because I think it can disturb further development of features. I'll come back to that later with an idea I had.

Ok, so:

On 11/29/2016 at 0:26 AM, mDuo13 said:

T8493 is correct -- transaction ordering is still deterministic (it has to be!) but is now harder to game. Basically the rule for how transactions from different addresses are ordered is now dependent on the contents or hash of the ledger itself (I don't remember exactly what) instead of being mostly based on alphabetizing the transaction hashes.

[...]

@mDuo13 and @T8493 say the order of transactions has changed. Making it more difficult / 'impossible' to do front-running this way. Is this really impossible? Is it possible only if you know the exact content of all entries in the upcoming next ledger?

On 11/24/2016 at 9:53 AM, tulo said:

This was changed, so the order is  should now be non-deterministic. For frontrunning I think now it is sufficient to spam lots of transactions with high fee, such that the other transactions are queued and your is executed before.

@tulo then says it is still possible by out-queuing the order by spamming high-fee transactions. Is the spamming such that the original order only makes it to the next ledger number? I guess this can work as long as the original order to front-run does not have LastLedgerSequence specified to the first ledger it is submitted in? (and thus can be prevented by always specifying a strict LastLedgerSequence policy? favouring not being front-runned over transaction execution ) 

Do you know if there are any bots actively using this for 'fun and profit'?

 

Other questions that I did not have time for yet to figure out myself (because I am not that familiar with the technical details of it).

  • Can orders already be submitted in batch mode? So: execute transaction if and only if other transactions in batch are also processed in this ledger number
  • Any good read (bit more technical than this ripple.com article, bit less technical than github link to the source code) on this escrow and payment channels?

Thanks a lot!

Soon I'll post a question/discussion/brainfart on native leverage trading in RCL.

Link to comment
Share on other sites

I think it depends on the role of front running.

In donch case it was an arbitrage bot and since circular payments are not supported (https://github.com/ripple/rippled/issues/1257) it was super important to frontrun because the risk is to succeed in one transaction and fail the second one, leaving the bot with useless (and probably lower priced) currencies.

If you want to front run big orders (the canonical case) it is risky because:

  • You can send the buy transaction with fill_or_kill flag --> no risks here
  • The sell transaction has to arrive before the big order --> here is the risk
  • If you don't want to have extra funds/liquidity, your buy transaction should arrive before your sell transaction --> this might not be a problem if you have high liquidity in both currencies

So I see the only possibility to use the fee queue, but I didn't have time to go in the details of the code and to test it. According to the description it seems possible.

17 minutes ago, Eik said:

I guess this can work as long as the original order to front-run does not have LastLedgerSequence specified

This is right but you can see the transaction, so you can front run only the ones without LastLedgerSequence as next ledger.

BTW one can always avoid to be front runned (in many ways), so you can use it only for orders that are not big and from people not using ad-hoc software, for example people manually sending orders to buy XRP.

Edited by tulo
Link to comment
Share on other sites

 

1 hour ago, Professor Hantzen said:

Just a heads up: the issue detailed in the "fun & profit" link was squashed by Ripple later in the year that the article appeared.  It took way too long for Ripple to fix it, but they did.  Essentially everything regarding "transaction ordering" in that article is obsolete.

Thanks. mDuo13 and T8493 mentioned indeed that front-running via that specific way of transaction ordering was not possible anymore. Do you have more info on how the ordering is done now? I.e.:

2 hours ago, Eik said:

[...] @mDuo13 and @T8493 say the order of transactions has changed. Making it more difficult / 'impossible' to do front-running this way. Is this really impossible? Is it possible only if you know the exact content of all entries in the upcoming next ledger? [...]

I couldn't find any Ripple Dev Centre entry for the method transactions are ordered now.

 

The rest I could find on Ripple Dev Centre btw:

Payment channel info:
https://ripple.com/build/amendments/#paychan
https://ripple.com/build/transactions/#paymentchannelcreate
https://ripple.com/build/transactions/#paymentchannelclaim
https://ripple.com/build/transactions/#paymentchannelfund

Escrow info
https://ripple.com/build/amendments/#escrow
https://ripple.com/build/transactions/#escrowcreate
https://ripple.com/build/transactions/#escrowfinish
https://ripple.com/build/transactions/#escrowcancel
https://tools.ietf.org/html/draft-thomas-crypto-conditions-02  <-- I'll have to dive into this one :D looking forward to it.

 

Link to comment
Share on other sites

As far as I understood it, the set of current transactions that would go into the next ledger is taken as a seed value to order them. Since all servers will only include the same transactions eventually this is deterministic, but hard to predict, since any transaction being submitted by anyone globally could change this.

Link to comment
Share on other sites

2 hours ago, Sukrim said:

As far as I understood it, the set of current transactions that would go into the next ledger is taken as a seed value to order them. Since all servers will only include the same transactions eventually this is deterministic, but hard to predict, since any transaction being submitted by anyone globally could change this.

To my understanding, this is correct, but I've only glanced tangentially over the C++ code for it.

Front-running via the transaction queue might be possible, but I think it's probably too costly to do profitably. The idea would be:

  1. Monitor for incoming transactions you might want to front-run (e.g. if you see a big order coming you buy and resell to that order at a higher price).
  2. When you see such a transaction, you force it to go into the transaction queue rather than the open ledger. This is easier said than done:
    1. You have to submit a whole lot of meaningless transactions with high transaction cost (adding up to a lot of XRP)
    2. The meaningless transactions would have to be sent from many different addresses, since any one address can have at most 10 transactions in the queue at once. Holding enough XRP to satisfy the reserve of all these accounts would be even more expensive.
    3. You can't kick a transaction out of an open ledger after it gets in, so you'd actually have to race the transaction across the net, submitting all your meaningless transactions to other rippled servers before they process the transaction you "saw" in step 1.
      1. This would be easier if the transaction you want to front-run got put in the queue rather than the open ledger at first. That would mean it's only possible to front-run transactions when the ledger is busy and the sender of the transaction didn't care to pay extra to get their transaction in fast. Keeping the open ledger full all the time would be crazy hard and expensive, basically DDoSing the RCL, because the open ledger's available spots increase when lots of transactions are making it in.
  3. While the transaction you want to front-run is in the queue, you submit the meaningful transaction(s) that are supposed to do the actual front-running business, paying higher transaction costs so you can cut in line.
  4. Then you wait for the transaction you front-ran to exit the queue... Profit! Unless the sender of the transaction took any countermeasures:
    1. If they notice they got queued and are worried about being front-run, they could submit a new equivalent of the same transaction with a higher transaction cost to get into the open ledger before your front-running transactions do.
    2. If they set the LastLedgerSequence aggressively enough, you don't have much time to do this; in fact, if their LLS is low enough their transaction will never get queued at all.

In short, front-running via the queue seems like it would only be possible in some rare situations, and it wouldn't be cheap. I wouldn't try it.

Link to comment
Share on other sites

2 minutes ago, mDuo13 said:

You can't kick a transaction out of an open ledger after it gets in, so you'd actually have to race the transaction across the net, submitting all your meaningless transactions to other rippled servers before they process the transaction you "saw" in step 1.

This was a doubt I have. If the transaction arrives to the server, then it is put on the open ledger, and after that a lot of transactions with higher cost arrive (in the same open ledger time frame), the transaction to front run can be queued afterward or it's over?

Link to comment
Share on other sites

3 minutes ago, tulo said:

This was a doubt I have. If the transaction arrives to the server, then it is put on the open ledger, and after that a lot of transactions with higher cost arrive (in the same open ledger time frame), the transaction to front run can be queued afterward or it's over?

If the transaction you want to front-run makes it into the open ledger of enough servers, it'll pass consensus and make it into the next validated ledger. At that point, you can hope that the canonical ordering randomly favors you, but that's not reliable. You can't kick a transaction "back into the queue" from the open ledger on one server. You might be able to race it to the open ledger on other servers.

Link to comment
Share on other sites

I would rather try to generate as many transactions as possible/profitable and hope that they get ordered in front of the target transaction. One issue is that by doing so, this also changes the ledger state and thus the random seed - so it might not be that useful to rely on heavy computations (which used to be possible before - just create a transaction with a smaller txid). It might be a better idea to just spam a lot of transactions and making as sure as possible that they will fail if they get scheduled AFTER the target transaction.

E.g. there is a chance for a 1 USD profit. You submit 100 transactions that either take this profit if they get scheduled before your target or fail completely, if they get scheduled after. Then it just matters how likely it is to actually get the money and how much fees you pay for the failing transactions. As long as this is still profitable, it would still make sense to try to front-run.

The transactions would likely need to be created and submitted very close to the 5 servers that matter, so I recommend scoping out Ripple Inc.'s preferred data centers etc. to get a good guess about their validator location.

Link to comment
Share on other sites

Mwokay. Clear enough.

13 hours ago, mDuo13 said:

[...] At that point, you can hope that the canonical ordering randomly favors you, but that's not reliable. [...]

@mDuo13 So when the to-be-front-runned transaction is in (transaction A). You submit 10 times a transaction from one wallet that only has the funds to execute one of them, giving you a serious edge in front-running it.

So in your example (seeing a big order coming, you buy and resell to that order at a higher price).

Assuming 5 equal buy orders (transaction B1) and 5 equal sell(-higher) orders (transaction B2), and your account initially only has funds to cover only one of the buy transactions[1].
Random canonical ordering means a chance of 50% to be processed earlier than any other transaction in the open ledger.
(EDIT: false statistics, see two posts below)
The chance that a B1-transaction gets processed before A equals 1-0.5^5=97%
The chance that a B2-transaction gets processed before A and after B1 equals ( I'll come back to this number tomorrow after some sleep and a coffee, but probably >90% ? )

The serious edge being somewhere >90%, and the remaining chance is either a completed buy order + open sell order or an open buy order. These can probably be canceled before the next open ledger closes or earlier.

[1] this process can be performed with multiple wallets in parallel.

Edited by Eik
false statistics, see two posts below
Link to comment
Share on other sites

1 minute ago, Sukrim said:

I would rather try to generate as many transactions as possible/profitable and hope that they get ordered in front of the target transaction. One issue is that by doing so, this also changes the ledger state and thus the random seed - so it might not be that useful to rely on heavy computations (which used to be possible before - just create a transaction with a smaller txid). It might be a better idea to just spam a lot of transactions and making as sure as possible that they will fail if they get scheduled AFTER the target transaction.

E.g. there is a chance for a 1 USD profit. You submit 100 transactions that either take this profit if they get scheduled before your target or fail completely, if they get scheduled after. Then it just matters how likely it is to actually get the money and how much fees you pay for the failing transactions. As long as this is still profitable, it would still make sense to try to front-run.

The transactions would likely need to be created and submitted very close to the 5 servers that matter, so I recommend scoping out Ripple Inc.'s preferred data centers etc. to get a good guess about their validator location.

How do you ensure that the transactions fails if they get scheduled after? The buy offer you can put a fill_or_kill, but the sell?

Link to comment
Share on other sites

12 hours ago, Eik said:

[...]
Random canonical ordering means a chance of 50% to be processed earlier than any other transaction in the open ledger.
The chance that a B1-transaction gets processed before A equals 1-0.5^5=97%
The chance that a B2-transaction gets processed before A and after B1 equals ( I'll come back to this number tomorrow after some sleep and a coffee, but probably >90% ? )

The serious edge being somewhere >90%, and the remaining chance is either a completed buy order + open sell order or an open buy order. These can probably be canceled before the next open ledger closes or earlier.

[...]

Ok, so what we learned after a coffee. Is that you should never trust anyone with statistics that didn't have a coffee yet. (I guess if you remove the coffee part, the statement becomes more true)

The corrected numbers are:
B1 before A: 83.3%
B1 before A and B2 in between B1 and A: 66.7%

The exact numbers are N/(N+1) and (N-1)/(N+1), where N is the number of buy and sell orders assuming they are similar (here N=5). I assumed in total 10 orders as I just read somewhere this is the maximum allowed orders per wallet per ledger.

  • If somehow you manage to increase this number by combining wallets or so (without increasing risk of losing money somehow), the odds  of successful front-running increases.
    E.g. N=10 yields 81.8% chance of successful front-running.
  • If you are ok not getting 100% of the available front-run reward, you can run multiple smaller wallets in parallel with small quantities of funds. In total, 2/3th of the wallets will take the reward and the other 1/3th not. Hoping you can mitigate your potential losses by canceling the orders on time or partially by fill_or_kill for the B1 buy order.

So if you are able to submit the yet unfunded B2 sell orders, this method should work for sure. I don't think this type of front-running can be ruled out due to the nature of the consensus process.

@tulo@Sukrim, you know this, are these unfunded B2 sell orders possible? I thought it was; only when consensus is reached, the actual transactions are verified and executed in the specified order?

Link to comment
Share on other sites

I think it is safer to have funds for both B1 and B2, such that you only have to frontrun with both A, without caring about the order of B1 and B2.

And BTW 10 is the limit for the transactions in the queue...actually you can send more, increasing the probability. I think it is worth to try it if you have time.

There will be 6 possibile orders (not considering other transactions):

  • A,B1,B2 --> Bad, you end up with a sell order
  • A,B2,B1 --> Bad, you end up with a sell order
  • B1,A,B2 --> Good, front runned
  • B1,B2,A --> Good, front runned
  • B2,A,B1 --> Bad, you end up with a sell order
  • B2,B1,A --> Good, front runned

 

so the important is to spam lots of B1, because that has to arrive before A. B2 can also arrive later (in case of no other transactions). With 10 transactions, the probability of front running is 99.9%. With 20 is 99.9999%....definitely worth it ;)

EDIT: I didn't take the coffee either :) the probability to fail is simply 1/N where N is the sum of your B1 transactions + 1. So for 20 your B1 the probability is:  4.7% to fail.

Edited by tulo
Link to comment
Share on other sites

1 hour ago, tulo said:

 

EDIT: I didn't take the coffee either :) the probability to fail is simply 1/N where N is the sum of your B1 transactions + 1. So for 20 your B1 the probability is:  4.7% to fail.

 

I always first scan every post on the coffee status to see if I have to take it serious or not. :D

More serious: are we talking about the same scenario? For me, example:

A = buy 1M XRP @ max. 1.05
B1 = buy 40k XRP @ max 1.04
B2 = sell 40k XRP @ 1.049

Orderbook, sell:
1.01, 10k XRP
1.02, 10k XRP
1.03, 10k XRP
1.04, 10k XRP
1.05, 1M XRP
etc

Of course the real transactions a bit neater, but you get the idea.

But now if order is B1,A,B2: bad, you end up with 40k XRP and an open sell order.

I guess you are right though about the enough funds for both orders. You will end up more often with an open sell order though, increasing risk.

Link to comment
Share on other sites

4 hours ago, Eik said:

Of course the real transactions a bit neater, but you get the idea.

But now if order is B1,A,B2: bad, you end up with 40k XRP and an open sell order.

I guess you are right though about the enough funds for both orders. You will end up more often with an open sell order though, increasing risk.

Yeah, exactly same scenario, and yeah, you are right. I was thinking that A was something like buy 1M and 40k, so that the B2 was fulfilled also after A....

If you don't have funds for both, the advantage is that you skip the bad situation A,B2,B1 but in all other good cases you have to order B1 and B2.

In general the open order B2 is not a big deal in my opinion, in particular if it happens 5% of the times. Because in the worst case your order gets filled but you can buy back and you lose the spread value. In most of the cases it doesn't get filled and you can delete it next ledger.

PS: I was thinking that sometimes I place orders with a price lot higher than actual price just to avoid making calculation (when trading manually). Next time I'll be careful because of possible front-runners :lol:

Link to comment
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now

×
×
  • Create New...