Application Development Meanderings

I was writing a scalaz-stream app the other day and could not figure out what .toStream really did. I looked at the unit test (the examples and unit tests are a great way to learn the library) and figured it out. When you create a Process0, a process that only has outputs, you need to eventually convert it to a Process[F[_], O]. Process0 means only outputs with zero inputs. Process1 means one input and one output. Here's the line that helped me: val source = emitAll(Seq(1,2,3)).toSource Once I saw that it made sense. emitAll() does not have an environment F[_] defined for it. It needs one in order to be able to run. So whenever you create a Process0 or Process1, you'll need to add an environment. If you are like me, I usually use the Task environment. Using toSource converts a Process0 to the default F[_] environment of task to create a Process[Task, Int] instead of a just a Process0[Int]. That's it!

When Mathematica reads a flatfile, it evaluates each field as a mathematica expression. I sometimes just want to read the entire file in as strings then perform some specific type conversion on columns. Here's a short way to import data is string fields. The idea is to provide a shell of the algorithm and then extensive customization through functions. This is a very functional approach to creating functions. However, the package is still composed of a giant function though versus a more clever, parsing state machine. You have options to customize line splitting, field/attribute processing, field level transformations as well a collection mechanism that allows you to collect specific input record for later use. For example, you can save the header line. The returned value is not just a data matrix, but an association with various import statistics and other collected information. You can find the package here in a  gist .

Update: I've created an electronic book to collect together scalaz-stream user notes:  My scalaz-streams User Notes Working on data streaming (many problems can be cast as data streams) is hard. Controlling synchronous and asynchronous behaviors easily and simply requires frameworks and code that is often is uncommon to most programmers, and hence, its hard to write the code while still retaining simplicity. Scalaz Streams (labelled sstreams in this article) help you manage complexity by providing a few fundamental abstractions. But I found the abstractions hard to use at first because I was not use to thinking in a model that sstreams uses. sstreams casts the problem as a state machine. There are 3 states and a "driver" that iterates through the states. Each state carries with it enough information to move to the next state. Each state is a "one step process" and so all states derive from the Process trait. The level of abstraction is pretty high which...

I forgot to add a full version of the NotesService that was discussed in a previous blog. Let's revisit the NotesService interface: /** * Primarily for larger notes/docs storage. * The API allows the note content to be retrieved * separately. Sometimes, we want to separate * out the larger, less changing content from * our core database into a separate component. A note * can be a picture, a spreadsheet, HTML, XML or * even just a text document. While times, * these types of documents are just stored in a * standard RDBMS database, there are other options * such as a document management systems or * file-system. * * This trait does not prescribe how to * create {{{Note}}} objects. Also, the {{{Note}}}, * {{{NoteId}}} and other types will probably escape * whatever object instantiates this object * so the types will need to be accessible to the outer * world for working with the service. That means * we will have to watch-out for path-depe...