[proposal][community pool spend] relayers who relay terra

Governance & Proposals
#1

[disclaimer]
I am currently a TFL employee. (I am not longer a TFL employee, effective 15-March-2022)
I run a validator, and
I also run some relayer nodes.

No funds from what I am proposing will go to me, in any way shape or form.
I am also not doing this on behalf of anyone except myself.
[/disclaimer]

The Problem

As most of you are aware, the terra chain has multiple bridges that allow people to transfer assets, one of those is called IBC, which allows people to transfer assets across the cosmos set of networks, like Osmosis, Juno, Secret, and of course Cosmos.
(see map of zones for a more precise list.)

In order to relay assets across the network, you need to submit transactions on each chain, and on terra that costs gas fees.

For the popular channels, this can cost a few thousand dollars a month each in fees alone for each participant. (not including hardware)

At the moment the people providing relaying services are doing this for the sake of the network, and have done so for the last ~6 months.

Due to the costs involved in doing this, some of the people providing relaying services have dropped out, and the remaining have taken turns keeping the relaying services up and running.

What I am asking for

I am asking for 3 things

  • Reimbursing relayers for the fees they have incurred so far
  • Paying relayers for the fees on a ongoing basis future basis until they can recoup their costs by other means
  • Paying relayers for a component of their infrastructure costs to run these services.

Reimbursement of fees

Relayer’s costs are visible on-chain, and easily auditable.
I am proposing that they are get reimbursed at 150% of their costs incurred until March1 2022. The 50% markup is a way of thanking them for stepping up.

Ongoing basis
We reimburse them at 110% of their costs ongoing.

Infrastructure costs
I propose we pay relayers to put towards their infrastructure costs. This would not be the full cost of the infrastructure required, but will help them.

How much am I asking for
I estimate the overall monthly ask for this will be ~20,000 UST/month. (120k/year).

I am asking for $1million UST to be deposited in Anchor earn, and the interest yield it generates be used to fund the relayers operational costs, with extra yield that it might generate get added to the principal.

This could be a smart contract that pays out the interest, so the principal is fully reclaimable by governance vote if the need arises. (and disclaimer, I have written a un-audited smart contract to do this kind of thing, but am hoping a fully audited solution is available, if not it would run fee-free)

I am also proposing that a multisig be formed on Apollo Safe with a mix of interested and independent parties to oversee the disbursement of the monthly yield, or via something like a cw-4 group contract.
Members of this group would be awarded a token amount for their effort out of the monthly yield.

I believe that this will help terra in the short-medium term until a more sustainable solution arrives, and is not lucrative enough that we will get swamped by 100’s of people who want to run relaying services.

I’m looking forward to other people’s feedback on this

9 Likes
#2

For context:

Currently, most IBC channels are only relayed by one entity. If this entity goes offline, the channel will be unavailable until the entity fixes the issue. This is the case for most Cosmos chains, not limited to Terra.

The reason for the lack of redundancy is that the IBC module currently cannot reject redundant packets if they arrive in the same block. E.g. If I send some OSMO to Terra, and there are two relayers relaying the Osmosis<>Terra channel, both of them will relay the packet containing my transfer to Terra. Only the 1st packet will take effect; the 2nd one will do nothing since it is redundant. However, both relayers have to pay gas fees for the transaction.

In most cases, to avoid paying the extra fee, relayers socially coordinate to have only one entity per channel. Remember that relayers are not incentivized. They pay these gas fees our of their own pockets. An incentivization mechanism can definitely help relayers creating redundancy which makes the channels more robust.

You might have noticed that Terra<>Osmosis channel went down for ~12 hrs today (3/17 19:30 – 3/18 08:00 UTC). This was due to a software bug on my relayer process (see image below). If there were another relayer serving as redundancy, we could have prevented the channel from going down completely. (Of course, I also need to improve the monitoring of my node so I can respond faster in such situations.

Screen Shot 2022-03-17 at 10.41.58 AM
Screen Shot 2022-03-17 at 10.41.58 AM1752×1534 381 KB

4 Likes
#3

My view is that any reimbursement is long overdue. It’s unfathomable that relayers currently have to fork this out of their own pockets.

I have a few questions (some of which are due to my lack of knowledge on IBC):

  1. Will the members of the multisig group also be responsible for calculating the expenses incurred by the relayers, or will it be calculated and submitted by the relayers themselves (and verified by the group on an ad hoc basis)?
  2. For the interest amount that $1 million UST can generate per month, how many relayers can your proposal support (and how many are actually needed)?
  3. As @larry mentioned, there is a need for redundancy. Will the proposed infrastructure costs pay the relayers for the redundancy as well? (I can imagine it also means the number of relayers that can be reimbursed will also be affected?)
3 Likes
#4

Direction of the proposal is great!

On a slightly different topic:
How do I know which validators are also relayers? Might want to add a way for non technical people to figure this out so that that they can delegate to validators that are also relayers.

1 Like
#5

Not all relayers run validators on the terra network.
It isn’t a requirement though.
If they wish to make it known they could simply sign a message (or put a memo in the transaction) identifying themselves, and that could be used as the basis of what you are suggesting I suppose.
But I think that’s a different topic than what the proposal is about

#6

  1. They would need a tool, and that is out of scope of this proposal, or they can verify the account transactions. @larry has a spreadsheet he may be willing to share.

  2. I believe 3-5 is a good number for each channel. Others will have their opinions.

  3. yes. the proposal would cater for 3-5 relayers, regardless of who ‘wins’ the transaction. if too many relayers start running, the amount of funds each individual relayer would get would diminish as well.

#7

Many relayers put memo in the transactions, for example:

Screen Shot 2022-03-18 at 3.06.59 PM
Screen Shot 2022-03-18 at 3.06.59 PM2170×120 17.1 KB

1 Like
#8

So as for next steps.

I believe we would need to find some independent volunteers (they don’t run a relayer, but are part of the terra community, and ‘trusted/known’ ) to who are willing to be part of the multisig.

I think what would be required from them is
a) create a signed message on their twitter with the wallet they plan to use (as @larry suggested)
b) about an hour a month of their time to approve the costs.

As these people will not have access to the actual endowment, just the interest generated I believe this level is sufficient.

(note: I am no longer a TFL employee)

3 Likes
#9

I 100% support this proposal.

#10

We support this as well. We at Lavender.Five support most channels in the cosmos, but avoid the Osmosis <> Terra connection because we simply can’t afford the cost of it. The current estimate for that single channel is in the vicinity of $10,000 USD per month, per relayer. Juno <> Terra is in the vicinity of $1,500 per month, per relayer.

When Osmosis <> Terra went down earlier this month, we were part of the force that helped stabilize it - but when the packets were flushed through, and stabilization confirmed, immediately disconnected. We would like to be able to support this channel, and other Terra channels, but it’s frankly too expensive.

One consideration that’s worth bringing up is also having validators coordinate a fee reduction. I proposed this over on Secret: Signaling Proposal: Reduce Validator Fees - Secret Governance - Secret Network to great success; it made users and relayers happy. Has this been considered here?

5 Likes
#11

oh…
I thought it was 4-5k/month.
If it is 10k, then we will need to budget for that x5, and as anchor earn’s interest rate decreasing, it means the initial ask will be going up as well.

Thanks for the input… I wouldn’t want this running out and having to go back for more in 3 months time.

Also… we don’t seem to have many people who want to step as as ‘independents’ ;-(

#12

Big +1 here

1 Like
#13

It costed $5,746 in Feburary. In March the tx volume increased, and my estimation is $6–7K.

#14

Am I correct in assuming we would need the capital to support this endeavor in order to be an “Independent” I’m just a little guy and don’t have much to offer in the way of money or hardware but I want to be more passionate and involved with supporting the ecosystem. Terra and IBC have been so incredibly influential in improving my life and it’s future prospects. If there’s something that us smaller fish can do I’d love to know what it is.

#16

so.
upon discussing with @larry,

we have a way to determine fees via GitHub - larry-stakehouse/ibc-stats: Stats on IBC message relaying.

once that is done, we will generate a proposal to send funds (@150%) to those relayer accounts.

As they were the accounts that incurred the costs, we shouldn’t need verification.

I might need some help generating the proposal itself (to ensure we can do a multi-send) transaction.

By doing it this way, we don’t need a secondary account to hold the funds to disburse, which removes one of the risks

2 Likes
#17

If you need a way to do a simple multi-send you can always try using Beluga. It’s one of Kujira’s dapps. It lets you send a token to multiple wallet addresses in one transaction for a 0.1% fee. They’re integrating smart contract functionality with it atm as well

1 Like
#18

I have put the initial code to generate the summaries here - GitHub - PFC-Validator/ibc-fees
I encourage others to check it does what it is supposed to do / matches other peoples estimations.

I have also checked in the raw & summary files into github alongside the code, so others can verify the process.

the summary for blocks 4985676 (IBC inception) to 70846738 (some time this weekend)
are available:

I’m going to ignore the ibc fee.

denom amount marked up @ 150%
usd 8,625 12,937
krw 30,378,914 45,56,8371
luna 1,920 2,880
sek 86,217 129,325

my only concern is that I am doing something incorrectly as there are over 90k accounts that have generated fees, and that’s more luna than I expected.

2 Likes
#19

I spoke to other relayers and I was counting ‘transfers’ which are not sent by relayers.
If we remove those we get ibc-fees/terra-fees-4985676-7102688-summary.csv at main ¡ PFC-Validator/ibc-fees ¡ GitHub

denom amount marked up @150%
uusd 415 622
krw 30,598,940 45,898,410
luna 914 1371
sek 89,480 134,220

with 51 different accounts doing relaying.

We are looking for one more volunteer to be a neutral party to join a multi-sig.
Once we have that I will put up a proposal to vote on.

4 Likes
#20

I support this initiative and am willing to participate in the multisig terra1lty8a7ur0t6l7qz56v5vh9cu3k3xeer2vev66v

#21

I have agreed to multi-sig on this proposal using wallet terra16l79rtfr2pjcax50ptxs69zaxzntvg433mtqpc