How We Scaled Meta Transactions At Origin – Origin Protocol

0 20


A few weeks ago Nick introduced the concept of meta transactions. To briefly summarize, meta transactions are how we are able to sponsor Ethereum transaction fees for our users. A key component of the meta transaction flow is our relayer, which sends transactions on behalf of our users to proxy contracts.

Relayer sends transactions on behalf of our marketplace DApp users

Meta transactions make it easier for sellers to onboard and encourages the use of our platform. We believe meta transactions are a major step forward in making our application more approachable, but our initial relayer prototype needed some improvements before releasing to production.

Primarily, we needed to be able to scale to support a larger volume of transactions. We expected our relayer to get swamped upon release since it could only process one transaction at a time. Our initial relayer would send a transaction, and wait for it to be mined before proceeding to process any more.

What we needed was an account abstraction that allowed the relayer to process transactions concurrently. Meet Purse, an open-source component of our relayer that provides an abstraction layer for firing off transactions into the ether(ha). Our relayer can take a subset of a transaction ({ to: "Alice's proxy", value: 0, data: 0x1234 }), give it to Purse along with a callback for when the transaction is mined, then purse will handle everything else.

// "Look at my simple interface!" - Purse
const result = await purse.sendTx({
to: proxy,
value: 0,
data: '0x01'
},
(receipt) => {
if (!receipt.status) {
console.error('Oh no!')
}
})
console.log('tx hash: ', result.txHash)

Since proxy contracts only care that a transaction is signed by the owner, and not which account actually sends the transaction, you can send from any available account. Purse was designed to be able to work with any number of accounts. You could stand it up with 1 or 1,000 senders. It uses the “standard” hierarchical deterministic wallet (See the ERC 84 discussion) implementation to generate accounts as needed from a single BIP39 mnemonic. This provided the flexibility we need to be able to scale as needed.

Purse is a component of the relayer

While being able to send concurrently from multiple accounts is a significant improvement, we can go even further with some internal nonce tracking. By tracking each account’s nonce, we can reliably send out as many transactions as we want without having to wait for them to be mined. Nonce tracking is a requirement because when you’re firing out multiple transactions blindly, the receiving nodes may or may not be aware of these pending transactions. It’s risky to rely on eth_getTransactionCount even with the pending argument since you may not be interacting with the same node on every request. Nonce tracking provides a higher degree of reliability and requires less trust in the JSON-RPC provider.

While we can fire off multiple transactions at once, it does have to be within the limitations of the transaction pool of our JSON-RPC provider. We can’t just send out a million transactions from a single account at once. go-Ethereum defaults to 16 per account, and Parity uses slightly more complex logic to decide when to garbage collect pending transactions. For this, we just added an arbitrary limit that we can adjust as we see fit. It’s easier and more reliable to scale with more accounts than going deeper into pending transactions.

Another goal of Purse was to simplify and automate the process of funding child accounts. Child accounts need to be funded so that they can pay for gas cost when sending transactions to the Ethereum network. We didn’t want to have to manually send Ether to all of the derived accounts. It would have been an operational burden for our team and potentially error-prone process. Purse takes funding into one master account and funds all the derived children without any manual intervention. Internally, Purse tracks the running balances for these child accounts and funds them when necessary automatically.

One of the largest challenges was to deal with the unreliability of the JSON-RPC providers. Even the best providers occasionally have issues that could (and have) crippled the relayer or put Purse into an unwanted state. We needed to be able to handle dropped transactions, 500 errors, connection timeouts, and even bugs in Purse. Our first task was to build solid monitoring and logging that we could rely on to detect any issue. We logged transactions details and state in a PostgresDB and instrumented the code with metrics logged in Prometheus. Then we built a set of operational dashboards in Grafana, as well as a business dashboard using Metabase so that we could have visibility into the behavior of the system. This allowed us to quickly detect and diagnoses edge case bugs as we rolled out Purse to production.

Trying to get favorable gas prices were also a challenge. To help with error handling and gas prices, we implemented our own custom web3.js provider, @origin/web3-provider built on web3-provider-engine but this may be a topic for another article. JSON-RPC error handling has been an ongoing challenge, but reliability is improving.

Nonce tracking also proved to need some special attention. Especially when a Kubernetes pod can be destroyed and recreated at any minute. Purse keeps track of the nonce in memory, in Redis, and as a last resort can reference the JSON-RPC provider if all else fails.

We also needed to keep an eye out for dropped transactions due to temporary gas spikes or time-based transaction pool garbage collection. Specifically, we check if the provider knows about the transaction( eth_getTransaction), if not, we’ll re-submit it to the provider. It doesn’t come up often, but this issue seems especially prevalent with some providers.

After a beta period, Purse is now a stable component part of Origin’s platform. As of this writing, our relayer has now processed over 9,000 transactions, with an average of 4 requests per user, created over 2,500 proxy contracts, 213 listings, consumed 1,069,786,764 gas, and saved our users almost 11 Ether in network fees.

We’ll continue to harden the relayer up to improve reliability and experience for our users as we get more usage data and discover more edge-cases.

We also want to keep improving Purse to scale as the platform grows. Right now the amount of child accounts Purse creates, and how many pending transactions are allowed per account are configured. As we grow, so will the load on the relayer. We will need improvements like auto-scaling of accounts, so Purse can derive a new account, fund it, then use it whenever it sees the load getting too high. We may also stand up multiple instances of the relayer with different master accounts. There’s a lot of potential for scaling this component of our infrastructure.

If you’d like to get involved in improving the Origin Protocol platform, please join us and say hi in #engineering in our Discord. We are always looking for talented contributors to help us achieve our mission of creating an open source and truly peer-to-peer commerce platform for everyone!

Learn more about Origin:

You might also like

Pin It on Pinterest

Share This

Share this post with your friends!

WhatsApp chat