Posts

Showing posts from May, 2017

univocity csv parsers, fs2 and scala

Univocity makes fast CSV parsers. I needed to marry them up to fs2 for stream processing in scala. First we need to create our univocity parser and begin parsing: valsettings=newCsvParserSettings() settings.setHeaderExtractionEnabled(true) valparser=newCsvParser(settings) parser.beginParsing(newFileReader(inputFile)) Now we need to connect up this source to fs2. Univocity is starting to implement java iterators which would make this easier, but we can create our own. Anytime we have an object that generates data, we need to think of using unfold, which means to unfold "something" into a stream of values: deftoStream[Record](parser: CsvParser) =Stream.unfold(parser) { p =>Option(p.parseNextRecord()).map((_, p)) } This works because unfold needs an Option with a tuple of a value to emit and the next "state". The initial state is the parser itself. Since Option(..) will return None when the argument is null and parseNextRecord() returns null when there a…