How to Synchronize files between unix and hdfs ?

How to Synchronize files between unix and hdfs  ?7

use these components in this order :  tFTPlist , tFTPget , tHDFSput

1- list files in your server, you can filter them. get the files in your talend server then push them to your HDFS distribution

2- If you want to bring only the missing files : use tFTPList to get all the file names from remote server and HDFS server, do an inner join between remote files and HDFS files and get the unmatched records, eg:
tFTPFileList–iterate–tFixedFlowInput1–main–tUnite–main–tMap–out1–>

Please contact us via this form below if you have any technical question.

 

Talend Expert can assist your organization with the adoption of Talend’s suite of products, Talend consulting for Open Studio, Integration Suite, the MDM suite, ESB and Big data suite, and in any stage of your project including Architecture and Planning, Design, Development and Deployment, Optimization of data integration jobs, Audit .. etc.

Leave a Reply