REST integration built-in OIC to read Large files with size more than 10MB

Processing large files in any integration requires a lot of effort and complex coding. Every integration platform has a limit size to read a file, similarly, the OIC service provides a limit of 10MB to read the file, if the size exceeds, then the below error will be encountered 

Error Reason: Translation Error.

Translation Error.

Error while translating message to native format.

Please make sure that the payload for the outbound interaction conforms to the schema and payload size does not exceed threshold. Error occured as File size exceeds maximum threshold size of 10 MB : Application Error Error Summary: ICS runtime execution error 

In this article, we will learn how to read the big-size file in OIC.  Note only XML and CSV formats are supported for larger file read operations. 

Here we will build an integration with a source FTP connector to read files containing sizes more than 10 MB  &  read them through the "Read File in Chunks" features and create multiple files  at the target FTP server 

Read File in Segments feature provides the capability to read bigger files in smaller segments to proceed further instead of reading at one go. 

Let's create a integration 

Go to Integrations , Click on Create 

Select Scheduled integration 

Provide the integration name as shown in the snippet 

Drop FTP connector on the workspace 

provide the endpoint name 

Select Operation as "Read a File "  and provide the directory from where you want to read the file. If the file size is more than 10MB then this integration will fail , let's check out  

provide the contents of your file by uploading the sample files

Create a sample EmpData.csv with columns like ID,FIRSTNAME,LASTNAME,DEPARTMENT and upload it 

Click on Done. 

Enable the identifier 

Activate the integration and run it 

Integration has been kicked off 

I have processed the file more than 10 MB, which is why it got ended in error. 

For more details look into below errors 

Now click on FTP adapter again and instead of selecting operation as "Read File Operation", select Download File .

Provide the input directory and provide the Download Directory, if the download directory is not existing, then ICS server will create itself 

Click on Done 

Now file downloaded by OIC needs to be staged 

drop the stage file operation on the pallette with the purpose to read the downloaded file 

Click on Next 

Select operation "Read File in Segments" 

Configure File reference 

File reference is nothing but the file which has been downloaded at the ICS server

Provide the content of file by uploading sample CSV file 

Click on Done, Here segment size is 200 which means it will create multiple  files with 200 records based on the large file 

Now stage file configuration done , now it time to provide target FTP connector where multiple needs to be created at server 

drop file connector onto the pallette 

Select operation as Write File 

Provide the content of file by uploading the sample files 

Click on Done. 

Drag the mapper to initiate the mapping activity 

map the columns 

Activate the integration

 As  per schedule, the integration will be called, and create files at the target directory