Importing Data Sources
The Sources panel is where you create and manage your connections to data sources.
Last updated
The Sources panel is where you create and manage your connections to data sources.
Last updated
Each data pod supports data collection and analytics on data taken from a variety of heterogeneous sources and formats. Here you connect to an harvest data from IoT routers and web sources.
To begin, select your data source and follow the steps indicated below:
Start your data pod:
Data import takes place in the Sources and Raw Data panels.
To establish a connection to your data source and start importing data, go to the Sources panel.
Your Sources dashboard contains an overview of all types of connections currently supported by the platform.
Each connection type has its own source card containing a list of established connections, dedicated in-platform documentation entries, and an Add Source button to create new connections. Expand each card to reveal the in-platform instructions and the list of the connections already created within a certain rubric.
From here, you can select your data source. Below is an overview of the individual source types.
Sensors or other publishers are connected to an IoT routing service with a standard internet URL.
The sensors publish messages on certain topics. To receive the messages that sensors publish on these topics, we first need to connect to the router and then subscribe to the topics on which the messages are published.
This is where you create connections to an IoT router within the Sources panel:
Data pods now only support connections to a WAMP router at this time. The WAMP protocol is a message protocol suited for IoT workload scenarios. The most common WAMP router is crossbar.io.
We will support additional routers/protocols such as RabbitMQ and Kafka soon. Stay tuned for future updates.
To create a connection, click on Add Source.
You need to specify a name for your connection, a URL, a realm, a description, as well as your preferred credentials type (username and password, certificate, ticket, or anonymous):
Click on Save to finalize.
After establishing a connection to an IoT router, you need to create subscriptions on one or more topics of the router to receive messages on that topic.
Once your connection has been added, it will be displayed in the overview panel where you have all of your IoT connections at a glance:
In the list of connections, you can start and stop connections or edit and create connections.
Once the connection has been established successfully, click on the play sign located right next to the newly created connection. This will change the status of your connection from STOPPED to CONNECTED:
You only receive IoT data if the connection is activated, at least one subscription for that connection is activated, and if a client publishes data on the subscribed topic.
Clicking on the edit sign opens the IoT connection editor where you can adjust your entries:
Click on Save to activate your changes. From here, you can also delete, stop, or start the connection by clicking on the designated buttons.
Once the connection to the IoT router has been established, you can begin with your data import into a Raw Table.
From the Sources overview panel, select your IoT connection and hit the Create Raw Table button that is displayed next to it.
You will be redirected to the Raw Data panel.
Continue to Data import from an IoT router to find out more about the next steps.
This is where you create new connections to a web API within the Sources panel:
Basic authentication is used to connect to the server.
Basic authentication is a simple authentication scheme built into the HTTP protocol. The client sends HTTP requests with the authorization header that contains the word ‘Basic’ followed by a space and a base64-encoded string username:password.
Click on Add Source to create a connection. The following window is displayed:
Here you enter the URL, a name for your connection as well as your username and password.
Also, you can use the query tool to generate a result from existing tables that can be passed as URL parameters. In our example, we have entered the URL https://restcountries.eu/rest/v2/name/{country}. In this case, the “country” parameter can be passed dynamically from a query result. Query: SELECT “c.country” as “country” FROM T_COUNTRY c WHERE continent = “Europe”.
Once your connection has been added successfully, it will be displayed in the overview Sources panel where you have all your web connections at a glance:
If you need to adjust your web poll rule or test your API, click on the edit icon to display the web poll editor:
Once done, hit Save to finalize.
From here, you can also test the API or delete the entry altogether by clicking on the designated buttons.
Once the connection to the web API has been established, you can begin with your data import into a Raw Table.
To do so, from the Sources overview panel, select your web connection and hit the Create Raw Table button that is displayed next to it.
You will be redirected to the Raw Data panel where you can start with your data import.
Continue to Data import from a web API to find out more about the next step.
Once you have learned how to establish a connection to an external data source, go to Raw Data Import to find out more about the different data import options.
If you have questions at this point or encounter any issues, get in touch with our support team.
