CustomErddap       EMSO ERIC - ERDDAP

ERDDAP > Files > Documentation

ERDDAP's "files" system lets you browse a virtual file system and download source data files. Hopefully, this is a familiar, easy system that you can use with your favorite web browser or, if you prefer, from a command line program like curl.

ERDDAP was designed around the idea that most datasets are huge, so most users just need or want a subset of the dataset that they are interested in (e.g., a smaller geographic area, a smaller time range, or not all of the data variables). But we understand that some users actually do want an entire dataset, or at least the subset which is found in a subset of the source data files. If that's you, then the "files" system may be for you. One advantage of the "files" system is that you can see each file's size (in bytes) and Last Modified time (Zulu time zone), so it is easy to see if a file has been changed.
 

Table of Contents

Click

To use the "files" system, just click. On any "files" web page, you can:

Other File Types

By default, a directory listing is returned as an HTML table on a web page. A user can request that a directory listing be returned in a different file format by appending any of these file extensions: .csv, .htmlTable, .itx, .json, .jsonlCSV1, .jsonlCSV, .jsonlKVP, .mat, .nc, .nccsv, .tsv, or .xhtml. For example, instead of this web page:
https://coastwatch.pfeg.noaa.gov/erddap/files/jplMURSST41/
you can request the directory listing as a .csv file:
https://coastwatch.pfeg.noaa.gov/erddap/files/jplMURSST41/.csv
 

WARNING! Different Metadata and Variable Names

For datasets available via ERDDAP's tabledap or griddap, ERDDAP administrators can set up ERDDAP to change a dataset's metadata and variable names on-the-fly so that you, the user, see an improved version of the dataset's metadata. But in "files", you will see the original metadata and variable names, so don't be surprised if they are different! If you aren't comfortable dealing with the different metadata and variable names, you might prefer using the dataset's Data Access Form instead.

Similarly, when you request a subset of data from one of ERDDAP's Data Access Forms, you can specify the file type (e.g., .nc, .csv, .json, .mat) that you want to receive in response. Naturally, the source data files available via "files" are just available in one file type. If you aren't happy with the source file's file type, you might prefer using the dataset's Data Access Forms instead.
 

Not All Datasets

Some datasets in this ERDDAP aren't available via the "files" system. Common reasons include: If the source files for a dataset that you want aren't available, you can email the administrator of this ERDDAP, nobody at example dot com, to request that they be made available, but there is usually a reason why they aren't already available.
 

Not FTP

We understand that some users might prefer that ERDDAP offer files via FTP instead of HTTP as is done by "files". Sorry. Hopefully, you'll be able to do what you need to do with the current "files" system.
 

Requesting Compressed Files

ERDDAP doesn't offer results stored in compressed (e.g., .zip or .gzip) files. Instead, ERDDAP looks for accept-encoding (external link) in the HTTP GET request header sent by the client. If a supported compression type (gzip, x-gzip, or deflate) is found in the accept-encoding list, ERDDAP includes "content‑encoding" in the HTTP response header and compresses the data as it transmits it. It is up to the client program to look for content-encoding and decompress the data accordingly. Requesting compression is optional, but compressed responses are often 3-10 times faster, so this is a big time savings if you are downloading lots of large files. (Note that there is no benefit to requesting compressed .png files since the files' contents are already compressed.)

Command Line Downloads with curl

If you want to download a series of files from ERDDAP, you don't have to request each file's ERDDAP URL in your browser, sitting and waiting for each file to download. Ways to use curl: ERDDAP+curl is amazingly powerful and allows you to use ERDDAP in many new ways. To install curl: Please be kind to other ERDDAP users: run just one script or curl command at a time.

Instructions for using curl are on the curl man page (external link) and in this curl tutorial (external link). But here is a quick tutorial related to using curl with ERDDAP:

View Media Files

For most common image and video file types, the "files" system will now display a '?' icon to the left of the filename. If you hover over that, you will see a popup window showing the image or an audio or video player.

Similarly, for a few audio file types (notably .mp3, .ogg, and .wav), you will see an audio control which allows you to listen to the audio file.

These preview features will only work for certain file types, in certain browsers, in certain operating systems. They rely on browser features, so they are largely out of our control.

Alternatively, if you click on the link for an image, audio, or video file, a viewer or player will open in a separate window. (If your browser asks you what you want to do with the file, tell it to handle the media file itself (not via other software), and tell it to remember this choice so that it will be used automatically in the future.)
 

"How can I work with these files?"

One of the main features of ERDDAP is that it allows you to download subsets of a dataset (via the dataset's Data Access Form) in whatever file format you want or make customizable graphs and maps via the dataset's Make A Graph web page (so you don't have to download data files or install any graphing software). These allow you to avoid having to work with the original source data files in file formats that you aren't familiar with and/or don't want to work with.

If you instead choose to download and work with the original source files offered by ERDDAP's "files" system, you have to figure out how you want to work with the files. Fortunately, there are lots of software tools for working with the various file types:

Byte Range Requests

Unlike requests for most of the other resources in ERDDAP, a request for a file from the "files" system (other than .nc and .hdf files) may include a "Range" request in the header which specifies a range of bytes to be returned, instead of the whole file. See Byte_serving (external link). This is used by some client software (for example, audio and video players in web browsers) to request chunks of the file instead of the whole file.

Accessing a remote file via byte ranges is often slow and inefficient. Sometimes it's worth it for reading small samples of remote files, notably audio and video files. But the more times you need access to the file, the more efficient it is to just download the file and then work with the local file.

ERDDAP's "files" system refuses all byte range requests to .nc and .hdf files, so don't even try to use Netcdf-java/c, ncview, Ferret, or other software tools to connect to .nc or .hdf files served by ERDDAP's "files" system as if they were local files. These requests are blocked because this approach is horribly inefficient and often causes other problems. Instead:


 
ERDDAP, Version 2.18
Disclaimers | Privacy Policy | Contact