1. Home
  2. Docs
  3. User Documentation
  4. Data upload

Data upload

 

Data can be uploaded to Datasets in openBIS Experiments and Objects (e.g., Experimental Steps). openBIS is agnostic of file formats and types.

First the Experiment/Object must be registered in the database, and then Datasets can be added to it. Data can be uploaded via the web user interface or via dropbox mechanism, as explained below. 

 

Upload via web interface

 

  1. Click the Upload button in the form, as shown below.

 

2. Select the dataset type (e.g. Attachment).

3. Fill in the relevant fields in the form. It is advisable to always enter a Name, because this is shown in the menu. If the name is not provided, the dataset code is shown.

4. Drag and drop files in the Uploader area or browse for files.

5. When uploading a zip file, users have the option to uncompress before import.

6. Save  

 

 

Note for MacOS users: the default MacOS archiver generates hidden folders that become visible in openBIS upon unarchive. To avoid this there are two options:

  1. Zip using  the following command on the command-line: zip -r  folder-name.zip folder-name  -x “*.DS_Store”
  2. Use an external archiver (e.g. Stuffit Deluxe).

 

Upload via dropbox

 

Web upload is only suitable for files of limited size (few GB). To upload larger data, openBIS uses dropbox scripts that run in the background (see Dropboxes). A default dropbox script is provided with the openBIS ELN-LIMS plugin, and the dropbox folder needs to be set up by a system admin. If this is available, users need to organise their data in a specific way:

 

Folder 1

     Folder 2

               data (can be single files or folders) 

 

Folder 1 needs to have a specific name that encodes the information of where the data should be uploaded to openBIS. Folder 2 can be named as wished by the user. If this folder is called “default” it will not be shown in the user interface, after upload. The name of Folder 1 can be generated from the ELN interface. From the page where you want to upload data, select Dataset upload helper tool for eln-lims dropbox from the More… dropdown and follow the instructions on screen. Place the correctly named Folder 1 containing your data in the eln-lims-dropbox folder. openBIS continuously monitors this folder and when data are placed here, they are moved to the final storage. The move happens after a predefined (and customisable) inactivity period on the eln-lims-dropbox folder.  

 

 

 

Dropbox with markerfile

 

In case of uploads of data >100GB we recommend to configure the eln-lims-dropbox-marker. The set up and configuration need to be done by a system admin. The process of data preparation is the same as described above, however in this case the data move to the openBIS final storage only starts when a markerfile is placed in the eln-lims-dropbox-marker folder. The marker file is an empty file with this name:  .MARKER_is_finished_<folder-to-upload-name>. Please note the “.” at the start of the name, which indicates that this is a hidden file. This file should also not have any extension.

For example, if the folder to be uploaded has the following name:

 

E+PBHOUMIK_PBHOUMIK+INTRA_BREED_GENETIC_VARIABILITY_OF_DOGS+INTRA_BREED_GENETIC_VARIABILITY_OF_DOGS_EXP_1+ATTACHMENT+my-data

 

The marker file should be named:

 

.MARKER_is_finished_E+PBHOUMIK_PBHOUMIK+INTRA_BREED_GENETIC_VARIABILITY_OF_DOGS+INTRA_BREED_GENETIC_VARIABILITY_OF_DOGS_EXP_1+ATTACHMENT+my-data

 

How to create the Marker file in Windows

 

You can create the Marker file in Windows using a text editor such as Editor. Any other text editor will  also work.

 

  1. open Editor.
  2. Save the file with a name such as .MARKER_is_finished_E+PBHOUMIK_PBHOUMIK+INTRA_BREED_GENETIC_VARIABILITY_OF_DOGS+INTRA_BREED_GENETIC_VARIABILITY_OF_DOGS_EXP_1+ATTACHMENT+my-data.
  3. The file is automatically saved with a “.txt” extension. This needs to be removed.
  4. Use the Rename option to remove the extension from the file.

 

How to create the Marker file on Mac

 

If you are not familiar with the command line, you can create a create an empty text file using for example the TextEdit application in a Mac. Any other text editor will also work.

 

  1. Open the TextEdit application and save an empty file with a name such as .MARKER_is_finished_E+PBHOUMIK_PBHOUMIK+INTRA_BREED_GENETIC_VARIABILITY_OF_DOGS+INTRA_BREED_GENETIC_VARIABILITY_OF_DOGS_EXP_1+ATTACHMENT+my-data.
  2. Save to any format. 
  3. You will get a message to say that files starting with “.” are reserved for the system and will be hidden. Confirm that you want to use “.”
  4. To show these hidden files, open the Finder and press Command + Shift + . (period)
  5. The file you saved before has an extension, that needs to be removed. If the extension is not shown in your Finder, go to Finder > Preferences menu, select the Advanced tab, and check the “Show all filename extensions” box.
  6. Remove the extension from the file.