How to import tabular data recursively and why?

Modified on Mon, 19 Dec 2022 at 04:56 PM

Importing data recursively enables to create a search pattern that will scan different folders and only select the relevant files. This can be extremly valuable when the data is spread across multiple folder. For example, you might have run hundreds of simulations, and each simulation creates a folder with results in different files, and you might want to select the same file for all simulations (e.g. results.csv).

This is something that you can do in the platform. If you create a new Import & Export > Tabular step, switch to Import Type Folders to enable the recursive import mode.

A new field called File Pattern appears:

Import Type “Folders” and new field “File Pattern”

In this mode you add folders instead of files to the datasets selection. Click on Add/Edit datasets. The file selection dialog opens. Folder entries have two clickable regions which behave differently:

  1. If you click on the folder icon at the beginning of the entry the folder is selected.
  2. If you click on the folder name you enter the folder and its content is shown. If you click.

Two regions for each folder entry: (1) select folder, (2) enters into folder


Selected folders


The importer goes through all selected folders and imports files which match the specified search pattern. The search pattern is specified in the field File Pattern.

How does the search pattern work?

Basic patterns (selecting all files in folders):

*

Wildcard, represents any character. All files in the selected folder(s) are imported.

*.csv

All .csv files in the selected folder(s) are imported.

*/*.csv

The importer imports all .csv files from the selected folder and from all folders one folder level below the selected ones.

**/*.csv

The importer goes through all subfolders below the selected folders and imports all .csv files which are found.

In all examples above you can replace .csv by any file ending you want to import.

You can improve the search by including further Strings in the search pattern. The Wildcard character can be used multiple times within one search pattern:

**/findme_*.csv

Importing all .csv files in all subfolders below the selected folder which start with the string findme_.

**/findme_*ABC*.csv

Like above but the files also need to include the string ABC.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article