3.1. PLMSynchro: New
PLMSynchro node uses an SQL database
Brief description: A new
data source node has been introduced for PLMSynchro that reads data directly from an SQL database
instead of from a CSV file. The goal was to replace or extend the previous CSV-based
data connection with a database connection.
The new node functions similarly to the existing
CSV source node.
Use cases:
Replacement for the
query
Staging table
and Watermark
Replacement for
Night Update: Update of ERP Data
Product Source
In the pipeline
Key Features
Support
for QODBC and QSQLITE as database drivers
Reading
records directly from database tables.
Configurable selection of:
Table
Fields
Key Field (Unique Key)
Filter Criteria
Sort Field
Support
for multilingual data imports using language-dependent field names (e.g.,
"FIELD1_deu")
Incremental Processing
To efficiently
process large amounts of data, a watermark function has been implemented:
The node
stores either the most recently processed date (CREATEDATE) or the
most recently processed ID.
In subsequent
runs, only newly added or modified records
are processed.
Alternatively,
a processing status can be maintained directly in the source table
(Pending, Success, Error).
Examples Provided
The issue included
several sample pipelines and SQL scripts:
Create a catalog structure from a
SQL Server table "Products".
Importing ERP data into existing
catalogs.
Example of
incremental updates using a watermark table and
time-driven data changes.
Benefits
for Users
Direct
connection to ERP, PDM, or staging databases without having to go through
CSV files.
Faster
and automatable data synchronization.
Support
for incremental imports reduces data volume and processing times.
Better
integration with existing enterprise databases and
ETL processes.[22]
