[New Fixture Format] - Create a new fixture format to validate block production

[marioevz]

Description

Create a new test type and fixture format that would help to verify the block building behavior of the clients, and indirectly, their mempool implementations to some degree.

The fixture type could be processed by the consume command and run using hive, by launching a client and performing many of the following steps in sequence but in the order dictated by the fixture:

• Send a transaction to the client using eth_sendRawTransaction
• Request a block from the client using engine_forkchoiceUpdatedV* and engine_getPayloadV*
• Re-org a block using engine_forkchoiceUpdatedV*

The fixture could also be processed by a client's internal test consumer (like evm blocktest or evm statetest).

The challenge of this format is that the ordering of the transactions of a block produced varies depending on the client, and therefore the verification is not straightforward.

To mitigate this, the fixture can contain rules on the verification of the transactions included in the payload, instead of expecting a single specific list of transactions or a block, for example:

• Mutually exclusive transactions: A list of transactions from which only one transaction can be included in the produced block, otherwise the test fails.
• Unsorted transactions: A list of transactions all of which must be included in the produced block in any order.
• Sorted transactions: A list of transactions all of which must be included in the produced block in the specified order.
• List of sorted transactions: A list of Sorted transactions of which any is an acceptable transaction order.

With these rules, the test consumer can evaluate the received block and either pass or fail the test.

We can explore the possibility to include all possible stateRoot combinations in the fixture, but this would make the test too fragile.

[jochem-brouwer]

Disclaimer: I think I just wrote about 2-3 semi-unrelated topic, treat this comment as a kind of brain-dump 😄 👍

I think the general idea for this is good, but I wonder what we are trying to test here. Clients/ELs can decide for themselves what transactions they are including and in what order. If a test mandates this, one is essentially testing if a block building strategy used by the client is following a certain ruleset. What would be an example here which could be tested? Should I see this as a setup for a test format to be used by for instance MEV operators?

In general, I think if a TxPool has X valid transactions where these do not exclude another (so no txs with the same sender and nonce), they are all willing to pay the base fee of the block, the nonce is correct, the sender has enough balance and the sum of the gas limit when these txs are executed (in any order) is less than or equal to the block gas limit, the test should check if all txs are included in the block. This could be in any order, of course following the default consensus rules (txs from same sender have increasing nonce in the tx list).

Test format: stateRoot check all combinations

We can explore the possibility to include all possible stateRoot combinations in the fixture, but this would make the test too fragile.

I don't think we should do this. The amount of different stateRoots possible with X txs is X factorial, so this scales super bad. If there's a test with a lot of transactions (for instance replaying a mainnet block) this becomes infeasible fast.

Race conditions
How are these tests going to account for certain behavior where parts of the client are not in sync? For instance, if a tx enters the txpool it will first get verified. Is the tx valid, does the sender have enough funds, and is the gas price high enough (and of course other things like nonce checks, etc.). If the tx just got received and we immediately request a block, the chances are low that the tx enters the block because it has not fully entered the mempool yet.

I'm not sure, but I also don't think that it's possible to inspect the mempool on most clients over RPC (and even if it is, there likely is no standard API for this), so it dos not seem possible to see if the tx has "succesfully" entered the pool (?). This thus could lead to false positives where txs are not included in the block, but the only reason is that they have not entered the mempool yet. When the test is re-ran, it could pass.

External test factors like timestamp
There are also situations we can think of where external factors play a role. For instance if we would test MEV and there is a tx which either pays coinbase a premium, or not, depending on the timestamp. This could thus lead to tests either passing or failing depending on the timestamp the test runs (likely need a way to fix the environment of the test, if this is not already implemented).

[marioevz]

What would be an example here which could be tested?

The main goal I'm thinking this test format should have is to verify that a client will not produce an invalid block.

For example, think of a test where we send one type-4 transaction from Account A with an authorization from Account B and zero nonce, then we send a transaction type-2 (or 0 or 1 or 3) from Account B also with zero nonce. The test then needs to verify that the transaction from Account B either gets included before the transaction from Account A, or never included at all.

I don't think we should do this. The amount of different stateRoots possible with X txs is X factorial, so this scales super bad. If there's a test with a lot of transactions (for instance replaying a mainnet block) this becomes infeasible fast.

I agree, even if we limit the cases to have only a few transactions, it can get bad.

I'm not sure, but I also don't think that it's possible to inspect the mempool on most clients over RPC (and even if it is, there likely is no standard API for this), so it dos not seem possible to see if the tx has "succesfully" entered the pool (?).

I was thinking of not querying the mempool at all, and try to verify using standarized eth_* or engine_* endpoints, so for example, if we know a transaction is invalid, we either expect it to be rejected when sent, or if we get no error, we request a block and verify it's not included in the built payload.

There are also situations we can think of where external factors play a role. For instance if we would test MEV and there is a tx which either pays coinbase a premium, or not, depending on the timestamp.

I think we should focus on protocol failures, rather than trying to game the mempool into including or rejecting a transaction based on fees. It will make this format somewhat limited but we can try at first with the simplest scenarios and then eventually expand to see if we could cover MEV or not.