CADC Data Web Service

The data web service is the primary file storage and retrieval interface for the CADC. This service can be used programmatically to upload files, download files, check file availability, and delete files in the context of a particular data archive.

Data Service Network Resources

Resource	Description
https://www.canfar.phys.uvic.ca/data/pub	Public data file transfer resource. /pub over HTTP does not gather user credentials, so if downloading a non-public file or uploading to a non-public folder, you will be redirected to https://www.canfar.phys.uvic.ca/data/auth and challenged for a userid/password.
https://www.canfar.phys.uvic.ca/data/auth	Authenticated data file transfer resource. This resource will challenge for a CADC userid/password for authentication and authorization.
https://www.canfar.phys.uvic.ca/data/pub	SSL data file transfer resource. A client certificate must be used to connect to this SSL-based resource. You will be authorized based on the credentials in the certificate.
https://www.canfar.phys.uvic.ca/data/transfer	Transfer negotiation endpoint for uploads and downloads.
https://www.canfar.phys.uvic.ca/data/transfer	Transfer negotiation endpoint that takes client certificates for authentication and authorization.
https://www.canfar.phys.uvic.ca/data/auth/transfer	Transfer negotiation endpoint that takes userid/password for authentication and authorization.
https://www.canfar.phys.uvic.ca/data/availability	Resource that can be used to check the availability of the data web service. Performing an HTTP get to this resource will produce an XML document describing the state of the service.

Data transfer techniques

Direct download - Perform an HTTP GET to /data/pub/<archive>/<file> and receive a redirect to the preferred download location.
Direct upload - Perform an HTTP PUT to /data/pub/<archive>/<file> and upload directly to the stream.
Negotiated download - HTTP POST a transfer document to /data/transfer (or /data/auth/transfer) and receive a transfer document with multiple download locations included.
Negotiated upload - HTTP POST a transfer document to /data/transfer (or /data/auth/transfer) and receive a transfer document with multiple upload locations included.

File identification URL elements for direct upload and download.

The following URL format is used to identify files in the data web service:

<https>://<data service resource>/<archive>/<fileID>

Element	Required?	Description
data service resource	Yes	The base URL identifying the resource of the data service. (See table below.)
archive	Yes	Identifies the data archive (Could be one of CFHT, HST, JCMT, VOSpace, etc.)
fileID	Yes	Identifies the file.

Documents for negotiated transfers.

An HTTP POST of a transfer document to /data/transfer (or /data/auth/transfer), where the transfer document has the format:

<?xml version="1.0" encoding="UTF-8"?>
<vos:transfer xmlns:vos="http://www.ivoa.net/xml/VOSpace/v2.0">
  <vos:target>ad:ARCHIVE/file</vos:target>
  <vos:direction>direction</vos:direction>
  <vos:protocol uri="ivo://ivoa.net/vospace/core#protocol" />
</vos:transfer>

and the following are valid attribute values

target - Has the format ad:<ARCHIVE>/ad:<fileID>
direction - Can be pushToVoSpace (upload) or pullFromVoSpace (download)
protocol - Can be ivo://ivoa.net/vospace/core#httpget or ivo://ivoa.net/vospace/core#httpsget or ivo://ivoa.net/vospace/core#httpput or ivo://ivoa.net/vospace/core#httpsput

will result in an response transfer document with URL endpoints for upload or download included.

<?xml version="1.0" encoding="UTF-8"?>
<vos:transfer xmlns:vos="http://www.ivoa.net/xml/VOSpace/v2.0">
  <vos:target>ad:ARCHIVE/file</vos:target>
  <vos:direction>direction</vos:direction>
  <vos:protocol uri="protocol">
    <vos:endpoint>upload/download URL 1</vos:endpoint>
  </vos:protocol>
  <vos:protocol uri="protocol">
    <vos:endpoint>upload/download URL 2</vos:endpoint>
  </vos:protocol>
  <vos:protocol uri="protocol">
    <vos:endpoint>upload/download URL N</vos:endpoint>
  </vos:protocol>
</vos:transfer>

The client should pick the top URL endpoint for the byte transfer and fallback to the other URLs if errors are encountered.

Authentication and Authorization

If trying to access a non-public file you will be required to authenticate either by a CADC User ID and password or through a client certificate over SSL. If the authentication (login) fails, you will get an HTTP 401 (Unauthorized) response. If you successfully authenticate but are not allowed to access to the file, you will get an HTTP 403 (Forbidden) response. If the file does not exist, you will get an HTTP 404 (Not Found) response.

Downloading a file

To download a file from the data web service, perform an HTTP GET on the URL that identifies the target file. For example, to download the file named I001B3H0.fits from the IRIS archive you would perform an HTTP get to URL:

https://www.canfar.phys.uvic.ca/data/pub/IRIS/I001B3H0.fits

If the GET is successful, you will received an HTTP response code of 200 and the file will be streamed to your client.

Checking for file availability and access

To simply check if a file exists and that you have access to the file, you can perform an HTTP HEAD request to the same URL that you would use to download the file. This HEAD request will allow you confirm its existence, your authorization, and to gather basic meta-data about the file.

To view the HTTP headers with curl, use curl --location --head or curl -L -I

With wget, use wget --server-response --spider

Headers prefixed with an X- are custom CADC headers; all others are standard HTTP 1.1 headers.

HTTP Header	Explanation
Content-Type	the mimetype of the file (optional: only present if type is known)
Content-Encoding	the type of encoding (typically compression) used (optional)
Content-Disposition	contains a suggested filename for clients that will write the file
Content-Length	size of the file as delivered
Content-MD5	The MD5 digest of the contents of the file.
Last-Modified	date of the last file modification (optional: not present when modified during delivery)
X-Uncompressed-Length	the size of the uncompressed file, in bytes (optional: not present when modified during delivery)
X-Uncompressed-MD5	The MD5 digest of the contents of the file when uncompressed. (optional: not present when modified during delivery)
X-CADC-Stream	The name of the Stream to use when performing a PUT request. (optional: Default Stream is used when none specified.)

HTTP GET options

Range options

Additional Header parameter(s) for use when downloading a file:

Number	Header	Value	explanation
zero or one	Range:	bytes=<x>-<y> (bytes x-y) bytes=<x>- (all bytes starting at x) bytes=-<x> (last x bytes)	Users may download a specific section of any file by using the Range request header. Please note that range requests are not compatible with cutout requests and will be ignored.

Examples

The following examples use the curl program. You must use the -g option to disable globbing so that curl ignores the [ and ] characters in the URL.

CFHT whole file download
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o

Example: curl --location-trusted -g -o 806045o.fits "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o"
CFHT download of byte range 200-500
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o

Example: curl --location-trusted -g -o 806045o.fits -H "Range: bytes=200-500" https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o"
CFHT download of remaing file starting at byte 500
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o

Example: curl --location-trusted -g -o 806045o.fits -H "Range: bytes=500-" https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o"
CFHT download of last 2000 bytes
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o

Example: curl --location-trusted -g -o 806045o.fits -H "Range: bytes=-2000" https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o"
CFHT download of last 2000 bytes
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o

Example: curl --location-trusted -g -o 806045o.fits -H "Range: bytes=-2000" https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o"
View meta-data (headers) of a CFHT image.
HTTP HEAD to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o

Example: curl -v --location-trusted -g --head "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o"

Cutouts

Additional URL parameter(s) for use when downloading a file to perform a cutout:

Number	Parameter	Value	explanation
one or more	cutout	[extension number][image section]	When requesting a file of type FITS, a number of cutout parameters may be included so that only these cutouts are retreived. We are using a subset of the CFITSIO image section specification for cutout specification. Please note that single cutout parameters can also be requested as a suffix in the file ID element of the URL.

Cutout syntax: examples

Image Section	Explanation
[1:512:2,2:512:2]	open a 256x256 pixel image consisting of the odd numbered columns (1st axis) and the even numbered rows (2nd axis) of the image in the primary array of the file.
[*,512:256]	open an image consisting of all the columns in the input image, but only rows 256 through 512. The image will be flipped along the 2nd axis since the starting pixel is greater than the ending pixel.
[*:2,512:256:2]	same as above but keeping only every other row and column in the input image.
[-,]	copy the entire image, flipping it along the first axis.
[3][1:256,1:256]	opens a subsection of the image that is in the 3rd extension of the file.

FITS Header retrieval

Additional URL parameter for use for downloading FITS header information:

Number	Parameter	Value	explanation
one or more	fhead	true	When requesting a file of type FITS, providing the parameter fhead=true will result in the download of the header information of the file only.

General Examples

Single Extension Cutout
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1]

Example: curl --location-trusted -g -o 806045o-cutout1.fits "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1]"
Pixel Coordinate Cutout
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHTSG/D3.IQ.R.fits[9979:10490,10573:11084]

Example: curl --location-trusted -g -o D3.IQ.R.9979_10490_10573_11084.fits "https://www.canfar.phys.uvic.ca/data/pub/CFHTSG/D3.IQ.R.fits[9979:10490,10573:11084]"
Extension and Pixel Coordinate Cutout
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1][1:100,1:200]

Example: curl --location-trusted -g -o 806045o-cutout2.fits "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1][1:100,1:200]"
Multiple Extension Cutout
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1]&cutout=[2]

Example: curl --location-trusted -g -o 806045o-cutout3.fits "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1]&cutout=[2]"
Multiple Extension Cutout with Pixel Coordinates
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1][10:120,20:30]&cutout=[2][10:120,20:30]

Example: curl --location-trusted -g -o 806045o-cutout4.fits "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1][10:120,20:30]&cutout=[2][10:120,20:30]"
Single Extension Cutout (Shortcut version)
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o[1]

Example: curl --location-trusted -g -o 806045o-cutout5.fits "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o[1]"
Extension and Pixel Coordinate Cutout (Shortcut version)
HTTP GET to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o[1][1:100,1:200]

Example: curl --location-trusted -g -o 806045o-cutout6.fits "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o[1][1:100,1:200]"
View meta-data (headers) of a CFHT image extension cutout
HTTP HEAD to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1]

Example: curl -v --location-trusted -g --head "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o?cutout=[1]"
View meta-data (headers) of a CFHT image extension cutout (Shortcut version)
HTTP HEAD to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o[1]

Example: curl -v -L -g --head "https://www.canfar.phys.uvic.ca/data/pub/CFHT/806045o[1]"

Data web service and file names

You can use the Content-Disposition returned in the getData HTTP header to easily get wget to write the downloaded file to the name the file is stored in the archive with by using its '--content-disposition' flag. Note that you might want to also use the 'no-clobber' option to avoid over-writing files you've already downloaded. There is no option for curl that is equivalent to wget's '--content-disposition' flag, but you could retrieve the HTTP header for the file, parse it for the content disposition and file name, then retrieve the file and saving it to that file name.

For URLs which specify a cutout, the suggested filename in the Content-Disposition header will include a extra part so that different cutouts from the same file will have different filenames. This extra part is intended to be somewhat human readable, though many characters are replaced with an underscore (_) to be generally more Internet and file system compatible. This extra part will be consistent between requests with the same cutout parameters.

Uploading a file

To upload a file to the data web service, you must have permission to write to the target archive. An upload is done by performing an HTTP PUT to the URL identifying the file, and supplying the file data in the accompanying input stream of the request. If successful, an HTTP 201 response code will be returned.

Upload example:

Uploading a file
HTTP PUT to: https://www.canfar.phys.uvic.ca/data/pub/CFHT/newFile

curl -T /path/to/newFile "https://www.canfar.phys.uvic.ca/data/pub/CFHT/newFile"

Download transfer negotiation

To negotiate a download, POST a transfer document to /data/transfer or /data/auth/transfer and receive a document with download URLs in preferred usage order.

Download transfer negotiation example:

Downloading a file

HTTP POST to: https://www.canfar.phys.uvic.ca/data/transfer

curl -d @mydoc -E mycert.pem -H "Content-Type: text/xml" "https://www.canfar.phys.uvic.ca/data/transfer"

where mydoc is:

        <?xml version="1.0" encoding="UTF-8"?>
        <vos:transfer xmlns:vos="http://www.ivoa.net/xml/VOSpace/v2.0">
          <vos:target>ad:CFHT/oldFile</vos:target>
          <vos:direction>pullFromVoSpace</vos:direction>
          <vos:protocol uri="ivo://ivoa.net/vospace/core#httpget" />
        </vos:transfer>

will result in a transfer document with download URLs:

        <?xml version="1.0" encoding="UTF-8"?>
        <vos:transfer xmlns:vos="http://www.ivoa.net/xml/VOSpace/v2.0">
          <vos:target>ad:CFHT/oldFile</vos:target>
          <vos:direction>pullFromVoSpace</vos:direction>
          <vos:protocol uri="ivo://ivoa.net/vospace/core#httpget">
            <vos:endpoint>http://uvic.canfar.phys.uvic.ca/transfer/CZhdXRoQ29kZT0wJnVybD1odHRwB1Ymxa8d2708b2c/oldFile</vos:endpoint>
          </vos:protocol>
          <vos:protocol uri="ivo://ivoa.net/vospace/core#httpget">
            <vos:endpoint>http://usask.canfar.phys.uvic.ca/transfer/ZXRoPWdldCZhdXRoQ29kZT0wJnVybD1odHRwJTNBJ/oldFile</vos:endpoint>
          </vos:protocol>
          <vos:protocol uri="ivo://ivoa.net/vospace/core#httpget">
            <vos:endpoint>https://www.canfar.phys.uvic.ca/transfer/WdleHA9MjAxNTA2MjQyMzAwNTEmYXJjPVRFU1QmaW/oldFile</vos:endpoint>
          </vos:protocol>
        </vos:transfer>

Upload transfer negotiation

To negotiate an upload, POST a transfer document to /data/transfer or /data/auth/transfer and receive a document with download URLs in preferred usage order.

Upload transfer negotiation example:

Uploading a file

HTTP POST to: https://www.canfar.phys.uvic.ca/data/transfer

curl -d @mydoc -E mycert.pem -H "Content-Type: text/xml" "https://www.canfar.phys.uvic.ca/data/transfer"

where mydoc is:

        <?xml version="1.0" encoding="UTF-8"?>
        <vos:transfer xmlns:vos="http://www.ivoa.net/xml/VOSpace/v2.0">
          <vos:target>ad:CFHT/newFile</vos:target>
          <vos:direction>pushToVoSpace</vos:direction>
          <vos:protocol uri="ivo://ivoa.net/vospace/core#httpput" />
        </vos:transfer>

will result in a transfer document with upload URLs:

        <?xml version="1.0" encoding="UTF-8"?>
        <vos:transfer xmlns:vos="http://www.ivoa.net/xml/VOSpace/v2.0">
          <vos:target>ad:CFHT/newFile</vos:target>
          <vos:direction>pushToVoSpace</vos:direction>
          <vos:protocol uri="ivo://ivoa.net/vospace/core#httpput">
            <vos:endpoint>http://uvic.canfar.phys.uvic.ca/transfer/CZhdXRoQ29kZT0wJnVybD1odHRwB1Ymxa8d2708b2c/newFile</vos:endpoint>
          </vos:protocol>
          <vos:protocol uri="ivo://ivoa.net/vospace/core#httpput">
            <vos:endpoint>http://usask.canfar.phys.uvic.ca/transfer/ZXRoPWdldCZhdXRoQ29kZT0wJnVybD1odHRwJTNBJ/newFile</vos:endpoint>
          </vos:protocol>
          <vos:protocol uri="ivo://ivoa.net/vospace/core#httpput">
            <vos:endpoint>https://www.canfar.phys.uvic.ca/transfer/WdleHA9MjAxNTA2MjQyMzAwNTEmYXJjPVRFU1QmaW/newFile</vos:endpoint>
          </vos:protocol>
        </vos:transfer>

Contact CADC for Assistance

For help and support with the CADC Data Web Service, please email cadc@nrc.ca