Fluctuating AWS Query Execution IDs causing Test Failures

[megetron3]

AWS query execution IDs are inconsistent and not deterministic even though using the reset mechanism (tests runs against proxy)

contest.py

def pytest_sessionstart(session):
    import requests
    proxies = {"http": "http://0.0.0.0:5005", "https": "http://0.0.0.0:5005"}
    requests.post("http://motoapi.amazonaws.com/moto-api/seed?a=42", proxies=proxies, verify=False)

# test.py
def test_athena_get_query_execution(set_athena_results):
    client = boto3.client("athena")

    response = client.start_query_execution(
        QueryString='SELECT * FROM your_table_name',
        QueryExecutionContext={
            'Database': 'your_database_name'
        },
        ResultConfiguration={
            'OutputLocation': 's3://your-bucket-name/query-results/'
        }
    )

    res = client.get_query_execution(
        QueryExecutionId=response["QueryExecutionId"]
    )

    assert res["QueryExecution"]["ResultConfiguration"][
               "OutputLocation"] == "s3://your-bucket-name/query-results/23e2fcb4-72d8-467d-894a-05e430b187ef.csv"

this test pass in the past but now i get:

E       AssertionError: assert 's3://your-bu...fa3b1799d.csv' == 's3://your-bu...430b187ef.csv'
E         
E         - s3://your-bucket-name/query-results/23e2fcb4-72d8-467d-894a-05e430b187ef.csv
E         + s3://your-bucket-name/query-results/bdd640fb-0667-4ad1-9c80-317fa3b1799d.csv

am i missing something?

[bblommers]

Hi @megetron3! I get the bdd... key as well when seeding Moto, and that is reproducible:

import boto3
import requests
from botocore.config import Config


def test_athena_get_query_execution():
    result()
    result()
    result()


def result():
    proxies = {"http": "http://0.0.0.0:5005", "https": "http://0.0.0.0:5005"}
    requests.post("http://motoapi.amazonaws.com/moto-api/seed?a=42", proxies=proxies, verify=False)

    proxy_config = Config(proxies={'http': 'localhost:5005', 'https': 'localhost:5005'})
    client = boto3.client("athena", "us-east-1", config=proxy_config, use_ssl=False, verify=False)

    response = client.start_query_execution(
        QueryString='SELECT * FROM your_table_name',
        QueryExecutionContext={
            'Database': 'your_database_name'
        },
        ResultConfiguration={
            'OutputLocation': 's3://your-bucket-name/query-results/'
        }
    )

    res = client.get_query_execution(
        QueryExecutionId=response["QueryExecutionId"]
    )["QueryExecution"]["ResultConfiguration"]
    assert res == {'OutputLocation': 's3://your-bucket-name/query-results/bdd640fb-0667-4ad1-9c80-317fa3b1799d.csv'}

Even when calling this code 3 times, the result is always the same.

Are there any other calls to Moto in between seeding Moto, and executing the tests? Because if another request also makes a call to random.uuid4 in the meantime, that request will have the b66..-uuid assigned, and the Athena result will get the next uuid (with whatever value).
So you'll always need the make the exact same sequence of requests in between calling /seed and get_query_execution, to get the same (reproducible) value.

[megetron3]

I understand the process now. After confirming the sequence of calls to random.uuid4, everything functions as expected!

By the way, is the only method to pre-determine these query execution IDs to run any function that utilizes random.uuid4, for instance, 100 times, and then store this data? I'm asking this because I'm interested in running parallel tests.

In other words, since you can't predict or set the UUID yourself, you must first invoke the function 100 times. This is particularly important when running parallel tests, as you wouldn't want to use the reset mechanism. You'd prefer a unique UUID for each test, which loads its own S3 file that mocks the Athena result file.

[bblommers]

Yes, correct. If you want to run tests in parallel, the UUID generation is no longer deterministic.

Just thinking out the top of my head, it's probably straightforward to add a feature that allows the user to configure a static QueryExecutionId to be returned, based on a specific input parameters. That way it can be deterministic on a per-test basis, regardless of how many other parallel tests are running.

If you want to open a new feature request for that, with some details on what a typical test/athena invocation looks like, I'd be happy to look into that further.

[megetron3]

ok will do. thank you!

[megetron3]

I attempted to consider this solution. It seems to function well for standard unit tests, however, it's not applicable for the moto proxy/server service.

The issue lies in the fact that the start_query_execution is scripted in a different codebase, which I don't have access to.

Perhaps a viable solution could be to recognize the test from within the start_query_execution mock function. However, I'm yet to come up with an effective solution. Do you have any suggestions?

Yes, correct. If you want to run tests in parallel, the UUID generation is no longer deterministic.

Just thinking out the top of my head, it's probably straightforward to add a feature that allows the user to configure a static QueryExecutionId to be returned, based on a specific input parameters. That way it can be deterministic on a per-test basis, regardless of how many other parallel tests are running.

If you want to open a new feature request for that, with some details on what a typical test/athena invocation looks like, I'd be happy to look into that further.

[bblommers]

What do you mean with to recognize the test from with the start_query_execution mock function?

It's difficult to tell whether this makes sense, without understanding the codebase/architecture, but you could also patch the start_query_execution-method to intercept the ExecutionId? That way you could skip the seed-function completely.