We're taking in a large number of CSV files from multiple publishers on a regular basis. Almost all of these files are inconsistent in some way, and therefore we've had to manually map many of them to the desired format.

We've now gathered a sufficient number of individual maps that we can, match almost all of the incoming files using a fuzzy match process (with a high score threshold to chuck out non-matches). Having made the correct matches, I now have to rename and save the csvs so that they can be input into the database, but this has got me stumped.

We've automated the mapping by scraping out the header as a whole line and matching to create a database table that stores these headers and their appropriate maps as long strings, as follows:

Headline | Mapline | Match_Score
"Col1, Col2, Col3" | "Map1, Map2, Map3" | 1.0
I had planned to cycle through each file and just do a 'replace in string' step to update the header and then save it, but then I realised that I can't dynamically set the search string in that step.

So, I've been looking at the metadata injection step, but I'm not sure how to make sure that I’m putting the right data into the right steps at the right time. I need to send three blocks of data into the metadata injector, (one for each step in the transformation that sits under the metadata injector) and I’m stumped on how to loop through each file as I use the metadata injector.

I was thinking that I might set a variable which pulls the right data from the table, but I’m not sure how to cycle that, and I’m thinking there must be a simpler way to do this. Any ideas much appreciated.