Home

Data Store

Whether your data is small or large—kilobytes to terabytes, or structured or unstructured—CyVerse's cloud-based Data Store can help you with your data storage needs.

About the Data Store

Based on a technology called iRODS, CyVerse's easy-to-use data hosting gives you great flexibility and control over your data. Choose from different interfaces—from web services to mountable file systems to high-speed command-line transfers—to connect your data to every CyVerse platform. To learn how to use the Data Store, see Manage Data.

Key features

Hosting your data in CyVerse's Data Store allows you to:

  • Use your preferred access method to upload, download, and access data from your computer to the Data Store.
  • Share your data instantly with other users.
  • Keep your data secure by setting permissions to keep your data private only to you, share it with specific collaborators, or make it accessible to the general public (so no account is required).
  • Take advantage of parallel data transfers for fast uploads and can handle partial transfers—so if a transfer of 1,000 files or one very large file is interrupted, the process can resume in the same place later.

Access methods

There are a number of different ways with which you can upload, import, and download private or public data in the Data Store. Most require a CyVerse account, but you can access public data anonymously using the Mirrors site (below) without an account, if you don't have one already set up.

Stampede Supercomputer at TACCIf you are more comfortable with a graphical interface, you can access your data from web-based platforms such as the Discovery Environment and Atmosphere.

If you prefer command line, you can use:

  • iRODS iCommands, which gives you the ability to transfer files quickly using multithreaded transfers. See Setting Up iCommands for steps on downloading and configuring iCommands for your operating system.
  • FUSE to mount a Data Store directory to a local directory, or view and navigate directories and directory contents, using the command line.
  • An API (for programmers).

If you prefer direct access, you can access your data directly using other interfaces in Python, C, Java, and REST API.

 

Concerned about data security?

We’ve kept your data security in mind. You can set permissions to keep your files private, or share them with specific collaborators, or even with the general public—no account required.

Curious about data allocation?

Each user has access to 100 GB of data storage, with the ability to request more allocation. Learn all about our data allocation policy here.

Want to access data but don’t have an account?

You can download some data files that have shared with the general public using several different methods. Find out more here.

Want to know more about data performance?

Below are the performance numbers for moving 100 GB of data from UC Berkeley to the CyVerse Data Store. To put some of these numbers into perspective, look at the transfer rate for the file annotation.MYI :

  • 22622 MB = ~22 GB
  • 367 sec = 6.1 minutes

This is the equivalent of transferring 5 DVDs (or one Blu-ray) in 6 minutes.

While this is over the internet2 backbone between UC Berkeley and UA Tucson, there are ~70,000 students and thousands more faculty, staff, and administration. All told, 102 GB of data was transferred between Berkeley and the CyVerse Data Store in 30 minutes — or 1 GB in 18 seconds. The following table compares other copying/transfer methods.

Source Destination Copy Method Time/GB
Berkeley Server (RAID10 7.2k)* CyVerse Data Store iRODS 18V sec
Mac Pro CD (50x) Mac Pro HD (7.2k) cp 5min: 20V sec
Mac Pro HD 7.2K USB 2.0 HD cp 36V sec
Mac Pro HD 7.2K internal cp 15V sec
Mac Pro UA Berkeley Server scp 2min: 30V sec
Mac Pro HD 7.2K USB2.0 Flash (Patriot XT) cp 48V sec
USB 2.0 Flash (Patriot XT) Mac Pro HD7.2K cp 30V sec

*The Berkeley machine is located in the UC Data Center with 1G to their network.

  • Berkeley's Server IP address: 169.229.192.155 (synteny.cnr.berkeley.edu)
  • CyVerse Data Store IP address: 128.196.172.132 (data.iplantc.org)

Ready to get started?