Skip to content

Commit 77204a5

Browse files
committed
Update download code to less confusing.
1 parent 1913470 commit 77204a5

4 files changed

Lines changed: 478 additions & 330 deletions

File tree

‎Article.md‎

Lines changed: 30 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -521,42 +521,56 @@ Result:
521521

522522
### Step 5: Downloading the file
523523

524-
That brings us to download the file. Based on the S3 ```file_url``` above, the actual file name is *Bulk-ESG-Global-Symbology-Organization-v1-Init-2023-11-26T16_04_11.525Z.jsonl.gz*. So you need to replace the escape character ```%3A``` with ```_``` (underscore) character.
524+
That brings us to download the file. You can download the bulk file using that S3 URL (**as is**). **Do not alter or make any changes to the URL text string**. It will cause unable to download or signature mismatch error.
525525

526-
**Note**: If you cannot download the file, please wait for a while and then retry download the file from the URL (```file_url```). Please do not flush the download requests. I am demonstrating with the [polling2](https://pypi.org/project/polling2/) library.
526+
**Note**:
527+
- If you cannot download the file, please wait for a while and then retry download the file from the URL. Please do not flush the download requests.
528+
- The code below set ```verify = False``` property in a ```requests``` library call to workaround LSEG's beloved ZScaler blocks a download request message. **Do not** set ```verify = False``` in a Production.
527529

528530
Python code:
529531

530532
```Python
531533
#step 5 - Download file
532534
import polling2
533535

534-
zipfilename = file_url.split("?")[0].split("/")[-1].replace("%3A","_")
535-
print(f'Downloading File {zipfilename} ...')
536-
537-
def test_result(_response):
538-
return _response.status_code == 200
539-
540536
try:
541-
bulkFile_response = polling2.poll(lambda: requests.get(file_url),
537+
print(f'Downloading File from {file_url} ...')
538+
response = polling2.poll(lambda: requests.get(file_url, verify= False),
542539
step = 10,
543540
poll_forever = True,
544-
check_success= test_result)
541+
check_success= lambda r: r.status_code == 200)
545542
except requests.exceptions.RequestException as exp:
546543
print(f'Caught exception: {exp}')
544+
```
547545

548-
if bulkFile_response.status_code == 200: # HTTP Status 'OK'
549-
print('Receive File Successfully')
550-
open(zipfilename, 'wb').write(bulkFile_response.content)
546+
ow you have downloaded the CFS bulk file stream in an application level. You can choose to save that file with whatever name you want.
547+
548+
If you need an actual file name of the file, it is available in S3 URL as follows:
549+
550+
```
551+
https://a206464-{bucket_name}.s3.amazonaws.com/XXX/YEAR/MONTH/DATE/{file_name}?x-request-Id={signature}.
552+
```
553+
Examples:
554+
- https://a206464-bulk-esg.s3.amazonaws.com/Bulk-ESG-Global-Symbology-Organization-v1/2023/11/26/*Bulk-ESG-Global-Symbology-Organization-v1-Init-2023-11-26T16%3A04%3A11.525Z.jsonl.gz*?x-request-Id=signature (an actual file name is **Bulk-ESG-Global-Symbology-Organization-v1-Init-2023-11-26T16_04_11.525Z.jsonl.gz**)
555+
- https://a206464-bulk-custom.s3.amazonaws.com/GE-11328/2025/06/12/*TM3_SIFMAIndex2025-06-12T14%3A00%3A00.000-04%3A00*?x-request-Id=signature (an actual file name is **TM3_SIFMAIndex2025-06-12T14_00_00.000-04_00**)
556+
557+
The actual file name has been replace a ```_``` (underscore) with ```%3A``` escape character, so an application needs to replace the escape character ```%3A``` with ```_``` (underscore) character to get an actual file name.
558+
559+
```python
560+
# Save the file locally.
561+
562+
if response.status_code == 200: # HTTP Status 'OK'
563+
zipfilename = file_url.split("?")[0].split("/")[-1].replace("%3A","_")
564+
print('Download File Successfully')
565+
open(zipfilename, 'wb').write(response.content)
551566
print(f'{zipfilename} Saved')
552567
else:
553-
print(f'RDP APIs: Request file failure: {bulkFile_response.status_code} {bulkFile_response.reason}')
554-
print(f'Text: {bulkFile_response.text}')
568+
print(f'RDP APIs: Request file failure: {response.status_code} {response.reason}')
569+
print(f'Text: {response.text}')
555570
```
556571

557572
Result:
558573
```bash
559-
Downloading File Bulk-ESG-Global-Symbology-Organization-v1-Init-2023-11-26T16_04_11.525Z.jsonl.gz ...
560574
Receive File Successfully
561575
Bulk-ESG-Global-Symbology-Organization-v1-Init-2023-11-26T16_04_11.525Z.jsonl.gz Saved
562576
```

0 commit comments

Comments
 (0)