File Import

File Format

For each data file that is imported into VDV, there needs to be a File Format that matches the data file format. Using the File Format editor VDV can import almost any kind of delimited text-based data file.

To Access: Data Source -> File Format

Click the button in the top-right corner to create a new File Format.

• Name: Identify the File Format with a descriptive name
• File Type: The file ending (typically csv or txt)
• Delimiter: Choose the correct Delimiter as in the file
• Decimal Point: Choose the correct Decimal point
• Owner: Choose which Owner the File Format belongs to. This is used so Users with access to specific Owners can only see File Formats that are connected to their Owners. If the File Format should be visible to all users then select the Fixed checkbox below.
• Header Line: If the file includes a header line with the variable name then select Yes.
• Header Line Number: Select in which line the Header Line is located.
• ID Column: For Sensor-Based data files select the column where the sensor ID number is located.
• Time Format: Can be in one or more columns. Add columns as needed and type in the format of the Timestamp

Sensor-Based Data Files

Create a new File Format for Sensor-Based files by setting ID Column to YES and set the ID Column Number to match the data file.

ID Name As Variable Prefix

Sensor-Based data file usually has multiple sensors and each sensor will become a site in the VDV System. This will result in multiple sites with the same variable names. The ID Name As Variable Prefix option inserts the sensor id name as a prefix for all variables.

An example of this would be:

A file with two sensors

POINT,Timestamp,X,Y,Z

• VDV01, 2021-09-20 13:00:00, 2000.8, 5090.6, -5.6
• VDV02, 2021-09-20 13:00:00, 2000.8, 5090.6, -5.6

If the ID Name As Variable Prefix enabled will yield variable names:

• Site: VDV01
• VDV01_X
• VDV01_Y
• VDV01_Z

and,

• Site: VDV02
• VDV02_X
• VDV02_Y
• VDV02_Z

If the ID Name As Prefix is not enabled then,

Site: VDV01

• X
• Y
• Z

and,

Site: VDV02

• X
• Y
• Z

File Locations

A File Location is a predefined locations where data can be imported from. It can be FTP, SFTP, Local network, or a local server.

To Access: Data Source -> File Location

Click the button in top-right corner to create a new File Location.

Local Directory

Adding a local directory is used to allow users to add new Data Sources from the web from the server running VDV. This can be used to add a directory where data is being collected.

• Name: Identify the File Location with a descriptive name
• Owner: Choose which Owner the File Location belongs to. Users with access to that owner can see and use the File Location
• Type: Choose Local
• Directory: Type in the path to the folder containing the data files. A list of all files and folders in that directory will be listed below

FTP Location

FTP Location allows users to import directly from FTP Servers. This allows data import from any external FTP/SFTP Server.

• Name: Identify the File Location with a descriptive name
• Owner: Choose which Owner the File Location belongs to. Users with access to that owner can see and use the File Location
• Type: Choose FTP /SFTP
• Address: Type in the address to the FTP Server (skip the ftp://)
• Type in Username and Password.
• Click Test Connection. If everything is working then a Successful notification will appear.

  

File Import

To Access: Data Source -> File Import

Click the button in the top-right corner to create a new File to VDV.

Data files can either be Incremental or Single where all data is appended to the same file.

Single File

• Start with typing in the Site Name that will appear in VDV and choose the Owner of the data.
• Choose Single File in File Structure
• Choose the correct format of the file to add
• Choose the Location of the file
• Open the folder and select the correct File (click on the EYE icon to see the data file content)
• Click Next

Incremental Files

• Start with typing in the Site Name that will appear in VDV and choose the Owner of the data.
• Choose Incremental File
• Choose the correct Format of the file to add
• Choose the Location of the file
• Type in File Prefix (Required). The import process will only match the incremental files that match the File Prefix.
• Open the Folder and select the correct File that matches the selected File Prefix. It does not matter which file you choose, they will all be processed in the import process.
• Click on the EYE icon to see the data file content
• Click Next

Sensor-Based Data

VDV supports Sensor Based data files, the user must first in file format make sure that the Sensor ID number is correct for the import file.

Adding Sensor-Based File

Adding Sensor-Based File via File Import is identical to any other file. Begin by filling out the default information as with any import in VDV. If the user selects a Sensor-Based Format the File Structure will automatically become Incremental since all Sensor-Based Files are required to be incremental.

Select the File for import and click Next

Confirm that the File has been parsed correctly

The Select variable name and unit window will be next, the only difference here is that when changing a variable name, it will change the variable name in all the sites created from a single file.

In the final step, an overview of the import you are about to create is displayed. If the File Format has the ID Name As Prefix option enabled the Alias Name will list the #ID#_name otherwise it will only be the variable name selected in the previous step.

Now the VDV System will begin importing data and if a new sensor appears in the data file VDV will automatically create a new site and import the data.

Deleting Sensor Based Data File

When deleting Sensor Data File a popup will open with information about the File Import and all the sites it will remove from The VDV System

Update Header

If the file that is linked to a Data Source has its headers changed, an admin must update the header of the file

The changes can be any of the following:

• Column(s) added to the data file
• Column(s) removed from the data file
• Column order changed in the data file
• Column(s) have been assigned to a new variable in the data file
• Or any combination of the above.

To navigate to the Update Header go to Data Source -> File Import and select the ... to open the Update header menu.

A Side by side comparison of the Current Configuration and the Latest Header will appear in the Update header menu.

Here the Humdity column was removed, "Wind" was changed to "Wind Speed" and a new column, "precipitation" was added.

To link the Wind Speed to the old Wind column, click the link symbol and select the "#4 - Wind" column.

After linking all renamed columns press Next.

Select a unit for all new headers and confirm that added and removed headers are correct.

Click Save and the updates the header will be made.