connection timeouts on certain order sizes

[m9brady]

In testing for the py-eodms-ddsapi package, I keep hitting connectiontimeouts for searches which are expected to return >100 images. I don't think it's a geometry vertex-count issue because I've used the same AOI for several other tests (with shorter time ranges) without seeing the same issue.

| EODMSRAPI | 2024-12-05 09:41:56 | Getting Collection information, please wait...
| EODMSRAPI | 2024-12-05 09:42:18 | Searching for RCMImageProducts images on RAPI
| EODMSRAPI | 2024-12-05 09:42:18 | Querying records...
| EODMSRAPI | 2024-12-05 09:42:18 | RAPI Query URL: https://www.eodms-sgdot.nrcan-rncan.gc.ca/wes/rapi/search?collection=RCMImageProducts&query=%28CATALOG_IMAGE.START_DATETIME%3E%3D%272024-11-05T00%3A00%3A00Z%27+AND+CATALOG_IMAGE.START_DATETIME%3C%3D%272024-11-18T00%3A00%3A00Z%27%29+AND+%28CATALOG_IMAGE.THE_GEOM_4326+INTERSECTS+POLYGON+%28%28-80.1486587+74.7931451%2C+-80.1541065+74.7938479%2C+-80.1922363+74.7801758%2C+-80.2126953+74.7494629%2C+-80.2102539+74.7036133%2C+-80.2206055+74.6570313%2C+-80.2627441+74.5844727%2C+-80.2777344+74.5815918%2C+-81.2262207+74.5666504%2C+-81.3404785+74.5535156%2C+-81.6071777+74.5023438%2C+-81.6249353+74.5000889%2C+-81.6137374+73.7977571%2C+-81.6136133+73.7944236%2C+-81.6133428+73.7910908%2C+-81.612926+73.7877592%2C+-81.6123631+73.7844293%2C+-81.6116543+73.7811017%2C+-81.6107997+73.7777767%2C+-81.6097995+73.774455%2C+-81.6086541+73.7711369%2C+-81.6073637+73.767823%2C+-81.6059285+73.7645139%2C+-81.6043489+73.7612099%2C+-81.6026251+73.7579116%2C+-81.6007577+73.7546195%2C+-81.5987468+73.7513341%2C+-81.5965928+73.7480558%2C+-81.5942963+73.7447851%2C+-81.5918576+73.7415226%2C+-81.5892772+73.7382688%2C+-81.5865556+73.735024%2C+-81.5836932+73.7317888%2C+-81.5806905+73.7285637%2C+-81.5775481+73.7253491%2C+-81.5742665+73.7221456%2C+-81.5708464+73.7189535%2C+-81.5672882+73.7157735%2C+-81.5635927+73.7126059%2C+-81.5597603+73.7094512%2C+-81.5557919+73.7063098%2C+-81.5516879+73.7031824%2C+-81.5474493+73.7000692%2C+-81.5430765+73.6969708%2C+-81.5385704+73.6938877%2C+-81.5339317+73.6908202%2C+-81.5291612+73.6877688%2C+-81.5242596+73.6847341%2C+-81.5192278+73.6817164%2C+-81.5140665+73.6787161%2C+-81.5087766+73.6757338%2C+-81.5033589+73.6727698%2C+-81.4978143+73.6698247%2C+-81.4921437+73.6668987%2C+-81.486348+73.6639924%2C+-81.4804281+73.6611062%2C+-81.4743849+73.6582405%2C+-81.4682195+73.6553957%2C+-81.4619327+73.6525723%2C+-81.4558965+73.6499329%2C+-81.4061523+73.6345215%2C+-81.3889868+73.6243701%2C+-81.3850626+73.6230388%2C+-81.3774011+73.6205027%2C+-81.3696322+73.617993%2C+-81.3617572+73.6155102%2C+-81.3537772+73.6130545%2C+-81.3456936+73.6106263%2C+-81.3375074+73.608226%2C+-81.3292199+73.6058539%2C+-81.3208324+73.6035104%2C+-81.3123461+73.6011958%2C+-81.3037623+73.5989104%2C+-81.2950823+73.5966546%2C+-81.2863074+73.5944287%2C+-81.2774389+73.592233%2C+-81.2684781+73.5900679%2C+-81.2594264+73.5879336%2C+-81.2502851+73.5858305%2C+-81.2410555+73.5837588%2C+-81.2317391+73.5817189%2C+-81.2223372+73.5797111%2C+-81.2128512+73.5777356%2C+-81.2032825+73.5757927%2C+-81.1936325+73.5738828%2C+-81.1839027+73.572006%2C+-81.1740944+73.5701627%2C+-81.1642091+73.5683532%2C+-81.1542484+73.5665776%2C+-81.1442135+73.5648362%2C+-81.134106+73.5631294%2C+-81.1239275+73.5614573%2C+-81.1136793+73.5598202%2C+-81.1033629+73.5582182%2C+-81.0929799+73.5566518%2C+-81.0825318+73.555121%2C+-81.0720202+73.5536261%2C+-81.0614464+73.5521673%2C+-81.0508121+73.5507448%2C+-81.0401189+73.5493588%2C+-81.0293682+73.5480096%2C+-81.0185617+73.5466972%2C+-81.0077008+73.545422%2C+-80.9967873+73.544184%2C+-80.9858226+73.5429835%2C+-80.9748083+73.5418207%2C+-80.963746+73.5406956%2C+-80.9526374+73.5396085%2C+-80.941484+73.5385595%2C+-80.9302875+73.5375488%2C+-80.9190494+73.5365765%2C+-80.9077714+73.5356427%2C+-80.896455+73.5347477%2C+-80.8851021+73.5338914%2C+-80.873714+73.5330741%2C+-80.8622926+73.5322958%2C+-80.8508394+73.5315567%2C+-80.8393561+73.5308569%2C+-80.8278443+73.5301964%2C+-80.8181455+73.5296745%2C+-80.827002+73.534668%2C+-80.8584961+73.5914063%2C+-80.8607422+73.6705566%2C+-80.848877+73.7212402%2C+-80.8228516+73.743457%2C+-80.7627441+73.7577637%2C+-80.6213867+73.767334%2C+-80.4123047+73.7654297%2C+-80.1202637+73.7070801%2C+-79.8893555+73.7015137%2C+-79.7582137+73.6841003%2C+-79.7575262+73.6845115%2C+-79.752615+73.6875451%2C+-79.7478348+73.6905952%2C+-79.7431864+73.6936615%2C+-79.7386705+73.6967435%2C+-79.7342879+73.6998408%2C+-79.7300394+73.7029529%2C+-79.7259255+73.7060793%2C+-79.7219471+73.7092196%2C+-79.7181047+73.7123734%2C+-79.7143991+73.7155401%2C+-79.7108308+73.7187192%2C+-79.7074004+73.7219104%2C+-79.7041087+73.7251131%2C+-79.700956+73.7283269%2C+-79.697943+73.7315512%2C+-79.6950703+73.7347857%2C+-79.6923383+73.7380298%2C+-79.6897475+73.741283%2C+-79.6872983+73.7445449%2C+-79.6849913+73.7478149%2C+-79.6828269+73.7510927%2C+-79.6808055+73.7543776%2C+-79.6789274+73.7576693%2C+-79.6771931+73.7609671%2C+-79.6756029+73.7642707%2C+-79.6741571+73.7675795%2C+-79.672856+73.770893%2C+-79.6716999+73.7742108%2C+-79.6706891+73.7775323%2C+-79.6698238+73.780857%2C+-79.6691042+73.7841845%2C+-79.6685305+73.7875142%2C+-79.668103+73.7908457%2C+-79.5842213+74.5430697%2C+-79.5839078+74.5464081%2C+-79.5837479+74.5497474%2C+-79.5837418+74.5530869%2C+-79.5838894+74.5564263%2C+-79.584191+74.5597649%2C+-79.5846465+74.5631023%2C+-79.585256+74.566438%2C+-79.5860196+74.5697715%2C+-79.5869371+74.5731022%2C+-79.5880085+74.5764297%2C+-79.5892339+74.5797535%2C+-79.590613+74.5830731%2C+-79.5921458+74.5863879%2C+-79.5938322+74.5896975%2C+-79.595672+74.5930013%2C+-79.597665+74.5962989%2C+-79.5998111+74.5995898%2C+-79.6021099+74.6028734%2C+-79.6045613+74.6061492%2C+-79.6071649+74.6094169%2C+-79.6099205+74.6126757%2C+-79.6128278+74.6159254%2C+-79.6158864+74.6191653%2C+-79.6190959+74.622395%2C+-79.6224559+74.6256139%2C+-79.6259661+74.6288217%2C+-79.6296259+74.6320177%2C+-79.633435+74.6352016%2C+-79.6373928+74.6383727%2C+-79.6414987+74.6415307%2C+-79.6457524+74.6446751%2C+-79.6501532+74.6478053%2C+-79.6547005+74.6509209%2C+-79.6593937+74.6540214%2C+-79.6642322+74.6571064%2C+-79.6692154+74.6601753%2C+-79.6743425+74.6632277%2C+-79.6796128+74.6662631%2C+-79.6850257+74.669281%2C+-79.6905804+74.672281%2C+-79.696276+74.6752627%2C+-79.7021119+74.6782255%2C+-79.7080872+74.681169%2C+-79.714201+74.6840927%2C+-79.7204525+74.6869963%2C+-79.7268408+74.6898791%2C+-79.733365+74.6927409%2C+-79.7400241+74.6955812%2C+-79.7468172+74.6983994%2C+-79.7537434+74.7011952%2C+-79.7608015+74.7039682%2C+-79.7679907+74.7067178%2C+-79.7753098+74.7094438%2C+-79.7827578+74.7121456%2C+-79.7903336+74.7148228%2C+-79.7980361+74.7174751%2C+-79.8058641+74.720102%2C+-79.8138165+74.7227031%2C+-79.8218921+74.7252779%2C+-79.8300898+74.7278262%2C+-79.8384083+74.7303475%2C+-79.8468464+74.7328413%2C+-79.8554027+74.7353074%2C+-79.8640761+74.7377453%2C+-79.8728653+74.7401546%2C+-79.8817689+74.742535%2C+-79.8907855+74.7448861%2C+-79.8999139+74.7472075%2C+-79.9091526+74.7494989%2C+-79.9185003+74.7517599%2C+-79.9279555+74.7539901%2C+-79.9375169+74.7561893%2C+-79.9471828+74.758357%2C+-79.956952+74.7604928%2C+-79.9668228+74.7625966%2C+-79.9767938+74.7646679%2C+-79.9868635+74.7667065%2C+-79.9970302+74.7687119%2C+-80.0072925+74.7706839%2C+-80.0176488+74.7726222%2C+-80.0280975+74.7745265%2C+-80.038637+74.7763964%2C+-80.0492656+74.7782317%2C+-80.0599817+74.780032%2C+-80.0707836+74.7817972%2C+-80.0816698+74.7835269%2C+-80.0926384+74.7852209%2C+-80.1036878+74.7868788%2C+-80.1148163+74.7885004%2C+-80.1260221+74.7900855%2C+-80.1373035+74.7916338%2C+-80.1486587+74.7931451%29%29%29+AND+ARCHIVE_IMAGE.PRODUCT_TYPE%3D%27GRD%27+AND+LUTApplied%3D%27Ice%27&resultField=CATALOG_IMAGE.THE_GEOM_4326%2CSENSOR_BEAM.SPATIAL_RESOLUTION&format=json
| EODMSRAPI | 2024-12-05 09:44:18 |  WARNING: Timeout Error: HTTPSConnectionPool(host='www.eodms-sgdot.nrcan-rncan.gc.ca', port=443): Read timed out. (read timeout=120.0); increasing timeout by a minute and trying again...
| EODMSRAPI | 2024-12-05 09:47:18 |  WARNING: Timeout Error: HTTPSConnectionPool(host='www.eodms-sgdot.nrcan-rncan.gc.ca', port=443): Read timed out. (read timeout=180.0); increasing timeout by a minute and trying again...
Traceback (most recent call last):
  File "<snip>\miniforge3\envs\py3-beta-eodms\Lib\site-packages\requests\models.py", line 974, in json
    return complexjson.loads(self.text, **kwargs)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "<snip>\miniforge3\envs\py3-beta-eodms\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "<snip>\miniforge3\envs\py3-beta-eodms\Lib\json\decoder.py", line 344, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "<snip>\miniforge3\envs\py3-beta-eodms\Lib\json\decoder.py", line 362, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<snip>\test_4.py", line 48, in <module>
    rapi.search(collection=collection, filters=filters, features=features, dates=dates)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<snip>\miniforge3\envs\py3-beta-eodms\Lib\site-packages\eodms_rapi\eodms.py", line 2889, in search
    return self._submit_search_query()
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "<snip>\miniforge3\envs\py3-beta-eodms\Lib\site-packages\eodms_rapi\eodms.py", line 2919, in _submit_search_query
    src_res = self._submit_search(show_log)
  File "<snip>\miniforge3\envs\py3-beta-eodms\Lib\site-packages\eodms_rapi\eodms.py", line 1220, in _submit_search
    r = self.rapi_session.submit(self.rapi_url)
  File "<snip>\miniforge3\envs\py3-beta-eodms\Lib\site-packages\eodms_rapi\rapi_requests.py", line 343, in submit
    return res.json() if as_json else res
           ~~~~~~~~^^
  File "<snip>\miniforge3\envs\py3-beta-eodms\Lib\site-packages\requests\models.py", line 978, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Script to reproduce

from netrc import netrc
from pathlib import Path
from time import sleep
from concurrent.futures import ThreadPoolExecutor

from eodms_dds import dds
from eodms_rapi import EODMSRAPI

this_file = Path(__file__)
eodms_user, _, eodms_pwd = netrc(
    Path('~/.netrc').expanduser()
).hosts['data.eodms-sgdot.nrcan-rncan.gc.ca']
rapi = EODMSRAPI(eodms_user, eodms_pwd)
# search options 
### the nested tuples/lists/dicts is hard for me to understand but I get that they're necessary 
### for multi-select filters. There HAS to be a more user-friendly way, like if a list of product types is 
### provided, the search-api must be smart enough to use the right operator. Probably needs logic to account 
### for range-type filters like incidence angle too
collection = "RCMImageProducts"
# these filters are a common use-case for me
filters = {
    'Product Type': ('=', 'GRD'),
    'LUT Applied': ('=', 'Ice'),
}
# this geojson is provided too
features = [
    ('intersects', str(this_file.parent / 'assets' / 'lancaster_gate_30km_buffer_clip.geojson')),
]
# these dates produce results of just over 100 granules
dates = [
    {
        "start": "20241105_000000",
        "end": "20241118_000000"
    }
]

out_dir = Path('~/Downloads/eodms-beta-test').expanduser()
out_dir.mkdir(exist_ok=True)

### quick-n-dirty download function for concurrent use later
def order_and_download(api_obj, item_ids):
    for item in item_ids:
        item_info = api_obj.get_item(collection=collection, item_uuid=item)
        while 'download_url' not in item_info.keys():
            sleep(10)
            item_info = api_obj.get_item(collection=collection, item_uuid=item)
        api_obj.download_item(out_dir)
    return

### the hit-count is nice to have for sanity-checks prior to "real" search queries
rapi.search(collection=collection, filters=filters, features=features, dates=dates)
results = rapi.get_results(form='full') # need to use full form to get uuids

### note how if the query params are adjusted (or even just the search is repeated with same params), the number
### of results just goes up (due to how rapi appends results rather than replaces)
### https://github.com/eodms-sgdot/py-eodms-rapi/blob/20d249f5660398b7201ae8e9c73ee65b5714a676/eodms_rapi/eodms.py#L2751

### ddsapi needs the uuids which are stored in a couple of spots but this one seems easiest to manipulate
### need to check for Nones because rapi returns None for some reason?
uuids = list(set([r['metadataFullName'].split('/')[-1] for r in results if r is not None]))
# download results
# really filthy concurrent method
n_workers = 8
# split uuids into roughly-equivalent batches
batches = [uuids[i::n_workers] for i in range(n_workers)]
# create api object for each worker
apis = [dds.DDS_API(eodms_user, eodms_pwd, environment='prod') for _ in range(n_workers)]
with ThreadPoolExecutor(max_workers=n_workers) as executor:
    futures = [executor.submit(order_and_download, api, batch) for api, batch in zip(apis, batches)]
    results = [future.result() for future in futures]

geojson file (rename to .geojson because GitHub didn't like that suffix): lancaster_gate_30km_buffer_clip.json

Package versions

dependencies:
  - asttokens=2.4.1
  - brotli-python=1.1.0
  - bzip2=1.0.8
  - ca-certificates=2024.8.30
  - certifi=2024.8.30
  - cffi=1.17.1
  - charset-normalizer=3.4.0
  - colorama=0.4.6
  - dateparser=1.2.0
  - decorator=5.1.1
  - exceptiongroup=1.2.2
  - executing=2.1.0
  - h2=4.1.0
  - hpack=4.0.0
  - hyperframe=6.0.1
  - idna=3.10
  - ipython=8.29.0
  - jedi=0.19.2
  - libexpat=2.6.4
  - libffi=3.4.2
  - libiconv=1.17
  - libmpdec=4.0.0
  - libsqlite=3.47.0
  - libxml2=2.13.5
  - libxslt=1.1.39
  - libzlib=1.3.1
  - lxml=5.3.0
  - matplotlib-inline=0.1.7
  - openssl=3.4.0
  - parso=0.8.4
  - pickleshare=0.7.5
  - pip=24.3.1
  - prompt-toolkit=3.0.48
  - pure_eval=0.2.3
  - pycparser=2.22
  - pygments=2.18.0
  - pysocks=1.7.1
  - python=3.13.0
  - python-dateutil=2.9.0.post0
  - python-tzdata=2024.2
  - python_abi=3.13
  - pytz=2024.2
  - regex=2024.11.6
  - requests=2.32.3
  - six=1.16.0
  - stack_data=0.6.2
  - tk=8.6.13
  - tqdm=4.67.1
  - traitlets=5.14.3
  - typing_extensions=4.12.2
  - tzdata=2024b
  - tzlocal=5.2
  - ucrt=10.0.22621.0
  - urllib3=2.2.3
  - vc=14.3
  - vc14_runtime=14.42.34433
  - vs2015_runtime=14.42.34433
  - wcwidth=0.2.13
  - win_inet_pton=1.1.0
  - xz=5.2.6
  - zstandard=0.23.0
  - zstd=1.5.6
  - pip:
      - click==8.1.7
      - geomet==1.1.0
      - py-eodms-dds==0.1.0
      - py-eodms-rapi==1.9.0