univocity csv parsers, fs2 and scala
Univocity makes fast CSV parsers. I needed to marry them up to fs2 for stream processing in scala. First we need to create our univocity parser and begin parsing: val settings = new CsvParserSettings () settings.setHeaderExtractionEnabled( true ) val parser = new CsvParser (settings) parser.beginParsing( new FileReader (inputFile)) Now we need to connect up this source to fs2. Univocity is starting to implement java iterators which would make this easier, but we can create our own. Anytime we have an object that generates data, we need to think of using unfold , which means to unfold "something" into a stream of values: def toStream [ Record ]( parser : CsvParser ) = Stream .unfold(parser) { p => Option (p.parseNextRecord()).map((_, p)) } This works because unfold needs an Option with a tuple of a value to emit and the next "state". The initial state is the parser itself. Since Option(..) will return None