Data Access API#

The imap-data-access repository provides a command-line utility and python package for interacting with the API programmatically. It is the preferred way to use the API.

IMAP Data Access

Users may also download, upload, and query via the REST API directly through the browser, or via curl commands. The REST API Specification section describes the various endpoints that are supported, and how to use them.

Note: Several sections and links begin with [WIP]. As development on the API is ongoing, this indicates that the full implementation of the functionality is yet to be completed.

The API can be accessed from the following URL [WIP]: https://api.dev.imap-mission.com

Command Line Utility#

To Install#

Run the following command to use the API CLI:

pip install imap-data-access

Base Command Arguments#

The following are base command arguments for the CLI:

imap-data-access -h # or
imap-data-access query # or
imap-data-access download # or
imap-data-access upload

Add the -h flag with any base command for more information on use and functionality.

Query#

To query for files, you can use several parameters: --instrument, --data-level, --descriptor, etc.

Further information is found in in the query -h menu. You can use parameters alone, or in combination.

Example Usage:

imap-data-access query --start-date 20240101 --end-date 20241231 --output-format json
# The following line is returned:
[{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe',
    'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'},
{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe',
    'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}]

Download#

To download files using the CLI tool, use the command download. The downloaded files will be placed in a data directory.

It is important to note that your working directory will be established as the default directory. I.e, the data directory will automatically be placed in this file path. Choose your working directory accordingly to suit your desires.

When downloading a file from the API, different folders within the data directory will be made to better organize the downloaded files. See the example path: data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts. The data directory and its structure is further described here: Data Directory

Example Usage:

imap-data-access download imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts

Upload#

Similarly, files can be uploaded to the API using the command upload.

When uploading files to the API, ensure these files are stored properly in a data directory (see the Data Directory section below for more information). Then, ensure your working directory is one level above data in order to properly upload files.

[WIP] Certain ancillary files can also be uploaded to the API. For more specific information regarding these files, visit Ancillary Files

Example Usage:

imap-data-access upload /imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf

Importing as a package#

Imap data access can also be imported and used as a python package.

Example Usage:

import imap_data_access

# Search for files
results = imap_data_access.query(instrument="mag", data_level="l0")
# results is a list of dictionaries
# Example: [{'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe',
# 'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105','version': 'v001', 'extension': 'pkts'},
# {'file_path': 'imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_v001.pkts', 'instrument': 'swe',
#'data_level': 'l0', 'descriptor': 'sci', 'start_date': '20240105', 'version': 'v001', 'extension': 'pkts'}]

# Download a file that was returned from the search
imap_data_access.download("imap/mag/l0/2024/01/imap_mag_l0_raw_202040101_v001.pkts")

# Upload a calibration file that exists locally
imap_data_access.upload("imap/swe/l1a/2024/01/imap_swe_l1a_sci_20240105_v001.cdf")

Configuration#

Data Directory#

The folder structure for data files within the IMAP SDC is rigidly defined, so the data access api will mimic that structure to make sure all data is stored in the same hierarchical structure as the SDC. This will enable seamless transition between a user’s local system and the SDC. This is only used for downloads.

A user’s root data location can be specified as an environment variable IMAP_DATA_DIR or through a configuration dictionary within the package itself (imap_data_access.config["DATA_DIR"]). If the IMAP_DATA_DIR variable is not set, the program defaults to the user’s current working directory + data/. The following is the directory structure the IMAP SDC uses.

<IMAP_DATA_DIR>/
  imap/
    <instrument>/
      <data_level>/
        <year>/
          <month>/
            <filename>

for example, with IMAP_DATA_DIR=/data:

/data/
  imap/
    swe/
      l0/
        2024/
          01/
            imap_swe_l0_sci_20240105_v001.pkts

Data Access URL#

To change the default URL that the package accesses, you can set the environment variable IMAP_DATA_ACCESS_URL or within the package imap_data_access.config["DATA_ACCESS_URL"]. The default is the development server (https://api.dev.imap-mission.com).

File Validation#

This package validates filenames and paths to check they follow our standards, as defined by the naming conventions. There is also a class available for use by other packages to create filepaths and filenames that follow the IMAP SDC conventions. To use this class, use imap_data_access.ScienceFilePath.

Usage:

science_file = imap_data_access.ScienceFilePath("imap_swe_l0_sci_20240101_v001.pkts")

# Filepath = /imap/swe/l0/2024/01/imap_swe_l0_sci_20240101_v001.pkts
filepath = science_file.construct_path()

Troubleshooting#

Network Issues#

SSL

If you encounter SSL errors similar to the following:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

That generally means the Python environment you’re using is not finding your system’s root certificates properly. This means you need to tell Python how to find those certificates with the following potential solutions.

Upgrade the certifi package
pip install --upgrade certifi
Install system certificates – Depending on the Python version you installed the program with, the command will look something like this:
/Applications/Python\ 3.10/Install\ Certificates.command

HTTP Error 502: BadGateway

This could mean that the service is temporarily down. If you continue to encounter this, reach out to the IMAP SDC at imap-sdc@lasp.colorado.edu.

FileNotFoundError#

This could mean that the local data directory is not set up with the same paths as the SDC. See the Data Directory section for an example of how to set this up.

REST API Specification#

Upload#

GET /upload/{filepath}#

Upload a science file to the SDC

Parameters:

filepath (string) – The full path to the file to upload, relative to the IMAP_DATA_DIR environment variable or the user’s current working directory. (e.g. {IMAP_DATA_DIR}/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts). The filename must be a valid IMAP Science or ancillary filename that follows the conventions described in our naming conventions.

Status Codes:

200 OK – Pre-signed URL for upload
400 Bad Request – Invalid or missing filename
409 Conflict – File already exists

Request Headers:

X-api-key – The API key for authentication. This key is used to authorize the upload request. (Optional for now).

PUT /upload/{filepath}#

Upload a science file to the SDC via pre-signed URL

Parameters:

URL (string) – The pre-signed URL for uploading the file. This URL is generated by the GET request to the /upload/{filepath} endpoint.
data (string) – The file data to be uploaded. This should be the binary content of the file.

Status Codes:

200 OK – Successful upload

Example Usage:

curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/upload/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts

Possible Responses:

{"statusCode": 200, "body": "https://sds-data-<aws_account_number>.s3.amazon.com/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts?<credentials-string>"}
{"statusCode": 400, "body": "Invalid filename. Expected - <mission>_<instrument>_<datalevel>_<descriptor>_<startdate>_<enddate>_<version>.<extension>"}
{"statusCode": 400, "body": "Invalid mission."}
{"statusCode": 400, "body": "Invalid instrument. Please choose from {'ultra-45', 'codice', 'glows', 'hit', 'lo', 'mag', 'swe', 'hi-45', 'idex', 'ultra-90', 'hi-90', 'swapi'}"}
{"statusCode": 400, "body": "Invalid data level. Please choose from {'l0', 'l1', 'l1a', 'l1b', 'l1c', 'l1d', 'l2'}"}
{"statusCode": 400, "body": "Invalid start date format. Please use YYYYMMDD format."}
{"statusCode": 400, "body": "Invalid end date format. Please use YYYYMMDD format."}
{"statusCode": 400, "body": "Invalid version format. Please use vxx-xx format."}
{"statusCode": 400, "body": "Invalid extension. Extension should be pkts for data level l0 and cdf for data level higher than l0"}
{"statusCode": 409, "body": "https://sds-data-<aws_account_number>.s3.amazon.com/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts already exists."}

Download#

GET /download/{filepath}#

Download a file from the SDC

Parameters:

filepath (string) – The full path to the file to download (e.g. imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts). The filename must be a valid science or ancillary IMAP filename that follows the conventions described in our naming conventions.

Status Codes:

302 Found – Successful download
400 Bad Request – Missing filename
404 Not Found – File not found

Example Usage:

curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/download/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01.pkts

Possible Responses:

{"statusCode": 302, "headers": {"Content-Type": "text/html", "Location": "s3://sds-data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01"}, "body": {"download_url": "s3://sds-data/imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-01"}}
{"statusCode": 400, "body": "No file requested for download. Please provide a filename in the path. Eg. /download/path/to/file/filename.pkts"}
{"statusCode": 404, "body": "File not found, make sure you include the full path to the file in the request, e.g. /download/path/to/file/filename.pkts"}

Query#

GET /query#

Query for file metadata

Query Parameters:

table (string) – The database table of interest (e.g. science or ancillary) for the query to perform against. The default option is the science table.
instrument (string) – The instrument of interest (e.g. mag). Supported instruments are listed in our naming conventions.
data_level (string) – The level of data product (e.g. l1a). Supported levels are listed in our naming conventions.
descriptor (string) – The descriptor of interest (e.g. burst). Supported descriptors are listed in our naming conventions.
start_date (string) – Search for all files with a start date on or after this time, in the format YYYYMMDD.
end_date (string) – Search for all files with a start date on or before this time, in the format YYYYMMDD.
ingestion_start_date (string) – Search for all files with an ingestion start date on or after this time, in the format YYYYMMDD.
ingestion_end_date (string) – Search for all files with an ingestion start date on or before this time, in the format YYYYMMDD.
version (string) – The version of data product in the format vNNN (e.g. v001). You can also choose to query --version latest in order to receive the most recent version of a file.
extension (string) – The file extension of interest (e.g. cdf). Supported extensions include pkts and cdf.

Status Codes:

200 OK – Successful query
400 Bad Request – Unsuccessful query

Example Usage:

curl -X GET -H "Accept: application/json" https://api.dev.imap-mission.com/query?instrument=swe&data_level=l0&descriptor=sci&start_date=20240105&end_date=20240105&extension=pkts

Possible Responses:

{"statusCode": 200, "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}, "body": [{"file_path": "imap/swe/l0/2024/01/imap_swe_l0_sci_20240105_20240105_v00-05.pkts", "instrument": "swe", "data_level": "l0", "descriptor": "sci", "start_date": "20240105", "end_date": "20240105", "version": "v00-05", "extension": "pkts"}]}
{"statusCode": 400, "headers": {"Content-Type": "application/json", "Access-Control-Allow-Origin": "*"}, "body": "<param> is not a valid query parameter. Valid query parameters are: ['file_path', 'instrument', 'data_level', 'descriptor', 'start_date', 'end_date', 'version', 'extension']"}

Data Access API#

Command Line Utility#

To Install#

Base Command Arguments#

Query#

Download#

Upload#

Importing as a package#

Configuration#

Data Directory#

Data Access URL#

File Validation#

Troubleshooting#

Network Issues#

FileNotFoundError#

REST API Specification#

Upload#

Download#

Query#

This Page