Data Source

File Format

For each data file that is imported into VDV there needs to be a File Format that matches the data file format. Using the File Format editor VDV can import almost any kind of delimited text-based data file.

To Access: Data Source -> File Format

Click the button in the top-right corner to create a new File Format.

• Name: Identify the File Format with a descriptive name
• File Type: The file ending (typically csv or txt)
• Delimiter: Choose the correct Delimiter as in the file
• Decimal Point: Choose the correct Decimal point
• Owner: Choose which Owner the File Format belongs to. This is used so Users with access to specific Owners can only see File Formats that are connected to their Owners. If the File Format should be visible to all users then select the Fixed checkbox below.
• Header Line: If the file includes a header line with the variable name then select Yes.
• Header Line Number: Select in which line the Header Line is located.
• Timestamp: Can be in one or more columns. Add columns as needed and type in the format of the Timestamp

File Locations

A File Location is a predefined locations where data can be imported from. It can be FTP, SFTP, Local network, or a local server.

To Access: Data Source -> File Location

Click the button in top-right corner to create a new File Location.

Local Directory

Adding a local directory is used to allow users to add new Data Sources from the web from the server running VDV. This can be used to add a directory where data is being collected.

• Name: Identify the File Location with a descriptive name
• Owner: Choose which Owner the File Location belongs to. Users with access to that owner can see and use the File Location
• Type: Choose Local
• Directory: Type in the path to the folder containing the data files. A list of all files and folders in that directory will be listed below

FTP Location

FTP Location allows users to import directly from FTP Servers. This allows data import from any external FTP/SFTP Server.

• Name: Identify the File Location with a descriptive name
• Owner: Choose which Owner the File Location belongs to. Users with access to that owner can see and use the File Location
• Type: Choose FTP /SFTP
• Address: Type in the address to the FTP Server (skip the ftp://)
• Type in Username and Password.
• Click Test Connection. If everything is working then a Successful notification will appear.

  

File Import

To Access: Data Source -> File Import

Click the button in the top-right corner to create a new File to VDV.

Data files can either be Incremental or Single where all data is appended to the same file.

Single File

• Start with typing in the Site Name that will appear in VDV and choose the Owner of the data.
• Choose Single File in File Structure
• Choose correct Format of the file to add
• Choose the Location of the file
• Open the Folder and select the correct File (click on the EYE icon to see the data file content)
• Click Next

Incremental Files

• Start with typing in the Site Name that will appear in VDV and choose the Owner of the data.
• Choose Incremental File
• Choose correct Format of the file to add
• Choose the Location of the file
• Type in File Prefix (Required). The import process will only match the incremental files that match the File Prefix.
• Open the Folder and select the correct File that matches the selected File Prefix. It does not matter which file you choose, they will all be processed in the import process.
• Click on the EYE icon to see the data file content
• Click Next

Update Header

If the file that is linked to a Data Source has its headers changed, an admin must update the header of the file

The changes can be any of the following:

• Column(s) added to the data file
• Column(s) removed from the data file
• Column order changed in the data file
• Column(s) have been assigned to a new variable in the data file
• Or any combination of the above.

To navigate to the Update Header go to Data Source -> File Import and select the ... to open the Update header menu.

A Side by side comparison of the Current Configuration and the Latest Header will appear in the Update header menu.

Here the Humdity column was removed, "Wind" was changed to "Wind Speed" and a new column, "precipitation" was added.

To link the Wind Speed to the old Wind column, click the link symbol and select the "#4 - Wind" column.

After linking all renamed columns press Next.

Select a unit for all new headers and confirm that added and removed headers are correct.

Click Save and the updates the header will be made.