Using the DVUploader tool

13/02/2025

When to use the DVUploader tool

The DVuploader tool can be a good alternative for uploading the files via the User Interface in the following cases:

• If you want to upload a lot of files;
• if you want to upload files with a complex hierarchical folder structure;
• when new files are being generated/added to a directory and Dataverse needs to be updated with just the new files;
• uploading of files needs to be automated, e.g. added to an instrument or analysis script or program.

Preparations

1. You will need Java version 8 or higher to be able to use the DVUploader. See this link to download Java: https://java.com/en/download/.
   
   
2. Then, click this link to download the DVUploader: https://github.com/GlobalDataverseCommunityConsortium/dataverse-uploader/releases/download/v1.2.1/DVUploader-v1.2.1.jar. (Clicking this link results in a direct download.)
3. There are two options for placing this jar file:
   
   • If you are not very familiar with using the command line, the following is recommended:
     Place the files you would like to upload in you home directory, together with the .jar file. This folder usually contains your name, for example, '.../Users/Laurahuisintveld'.
   • If you know how to change your working directory via the Terminal/Command prompt, you can store the .jar file into the directory containing a subdirectory with the files intended for upload.
     
     
4.  Find out the DOI of the dataset you would like to add files to. If there is no dataset yet, create one first. You will need the DOI in this format: doi:10.34894/ADBKTN
   
   
5. You will need your DataverseNL API-key. You can create and find your API Token in the User Interface only. Login into DataverseNL, click on your account name in the top navigation bar and choose 'API Token' from the drop-down list. Note: this token will automatically become invalid after a year.
   
   

Terminal (Mac)/Command Prompt (Windows)

1. Open the Command Prompt (for Windows) or Terminal (MacOS). This program is standard available on your Windows/Mac computer. For Windows, type in 'cmd' in your search box. For MacOs, Search for 'Terminal' in the 'Spotlight Search' (top navigation bar) or in the Launchpad.
2. If you have placed your .jar file not in your 'home' directory, change the directory first in your Terminal/Command Prompt. Otherwise, skip this step.
3. You need to adjust this line to suit your needs:

java -jar DVUploader-v1.2.1.jar -key=<api key> -did=<dataset doi> -server=<server URL> -uploadviaserver -recurse <dir or file names> 

• Replace <api key> with your DataverseNL API Token. (Remove the < > brackets from the example as well.)
• Replace <dataset doi> with the DOI of your dataset. (Remove the < > brackets from the example as well.)
• Replace <server URL> with https://dataverse.nl (or use https://demo.dataverse.nl to try it out on the demo environment first.) (Remove the < > brackets from the example as well.)
• Replace <dir or file names> with the name of your folder you would like to upload, or paste a list of files names instead. The name of the folder can not contain a comma. It can also be easier to remove spaces and replace the with an underscore '_'.  If you are listing single files, also include the file extension. (Remove the < > brackets from the example as well.)
• If you don't want to keep the folder structure, remove '-recurse', from this line.

Example:

java -jar DVUploader-v1.2.1.jar -key=dc00d47f-5a49-4a82-93c6-458763a32a8e -did=doi:10.80227/PDVNL/KHN6P4 -server=https://demo.dataverse.nl -uploadviaserver -recurse Testdocumenten 

(This API-Token has been made invalid.) In the example above we are uploading the folder 'testdocumenten' with all its subdirectories to the dataset with doi 10.80227/PDVNL/KHN6P4. 

4. Paste your adjusted line into the Command Prompt/Terminal.

The DVuploader will check if all the required values have been entered, and if so, the upload process will start. It checks for each file whether it already exists in the dataset, and if not, the upload is carried out. As soon as the DVUploader is completely ready, you will see:

***Execution Complete.***

5. A log file will be automatically created and stored in the same location as the .jar file. You can check the log (search for 'error') to see if any errors occurred.

6. Additionally, you can check the dataset in the DataverseNL UI. The files will be added to the DRAFT version of your dataset. So a new DRAFT version of your dataset has been created, or the files were added to the already existing DRAFT. 

7. For MacOs Users: Sometimes hidden files with the .DS_Store extension are uploaded as well if files were compressed. Check after the upload if your datasets contain any and remove them before publishing. 

Troubleshooting

The command you paste into the Terminal/Command Prompt is very sensitive for typing errors, and should not contain hidden characters or lay-out information that sometimes are included while copying and pasting text.
Once you run the command the DVUploader tool will first check your arguments. You will see this information below this header:  ***Parsing arguments:***

If you see something like this,

Using apiKey: xxxxxxxx-8xx5-4xxf-8xxf-3xxxxxxxx38f

Adding content to: doi:10.34894/OMPYOQ

Using server: https://dataverse.nl

Request to upload: -uploadviaserver -recurse Nurses_Scan_Sessions_3

You can see that the tool could not find your arguments '-uploadviaserver' and
'-recurse'. Instead, it thinks these are the names of folders you would like to upload. This would be a correct response:

Using apiKey: xxxxxxxx-8xx5-4xxf-8xxf-3xxxxxxxx38f

Adding content to: doi:10.34894/OMPYOQ

Using server: https://dataverse.nl

Will not use direct upload of files (not recommended for large files)

Will recurse into subdirectories

Request to upload: Nurses_Scan_Sessions_3

As a solution, try to copy and paste without copying lay-out. Use the 'kladblok' for example to avoid this. Or, if the problems persists, type the command yourself by hand and avoid pasting at all. 

Adapted from:

https://github.com/GlobalDataverseCommunityConsortium/dataverse-uploader/wiki/DVUploader,-a-Command-line-Bulk-Uploader-for-Dataverse (27/02/2025)