How to import tabular data recursively and why?

Importing data recursively enables to create a search pattern that will scan different folders and only select the relevant files. This can be extremly valuable when the data is spread across multiple folder. For example, you might have run hundreds of simulations, and each simulation creates a folder with results in different files, and you might want to select the same file for all simulations (e.g. results.csv).

This is something that you can do in the platform. If you create a new Import & Export > Tabular step, switch to Import Type Folders to enable the recursive import mode.

A new field called File Pattern appears:

Import Type “Folders” and new field “File Pattern”

In this mode you add folders instead of files to the datasets selection. Click on Add/Edit datasets. The file selection dialog opens. Folder entries have two clickable regions which behave differently:

1. If you click on the folder icon at the beginning of the entry the folder is selected.
2. If you click on the folder name you enter the folder and its content is shown. If you click.

Two regions for each folder entry: (1) select folder, (2) enters into folder

Selected folders

The importer goes through all selected folders and imports files which match the specified search pattern. The search pattern is specified in the field File Pattern.

How does the search pattern work?

Basic patterns (selecting all files in folders):

In all examples above you can replace .csv by any file ending you want to import.

You can improve the search by including further Strings in the search pattern. The Wildcard character can be used multiple times within one search pattern: