Posts

Showing posts from 2018

abstracting a react data fetcher in scala.js with cats, cats-effect

abstracting a react data fetcher in scala.js with cats, cats-effectProblemI put together a library for my small scala.js reasonreact-like facade scalajs-react that reflects the ReasonReact react component interface. When you use this facade to build web user interfaces you still need to fetch data. You have many choices for building a fetcher component including using an application state management library like redux/mobx or scala.js-based diode.The reasonreact component API includes a “reducer” built into every component so let’s just use that capability since its builtin and simple. I’ve found that with the builtin reducer, the need for global application state management is greatly reduced. If we can use a solid effects library then we should be all set. react@next has suspense and lazy loading. Unfortunately, these are not available in scala.js and may never be efficient in scala.js because scala.js cannot be easily compiled into separate “modules” without incurring larger runtim…

Ok, I think John was right about IO bifunctor..

Ok, I think John was right about IO bifunctor..A while ago there a post on how IO in scalaz had both a F and an E to reflect the error type that is more specific than Throwable. Cat has MonadError and ApplicativeError that assumes the error is a Throwable. That has alot of implications all up and down the APIs. Just as F has to be everywhere, once you want to customize the error to be more specific than Throwable or not related to Throwable at all much like as described in https://typelevel.org/blog/2018/11/28/http4s-error-handling-mtl-2.html, you have to fix all the APIs all the way down.That means the post http://degoes.net/articles/bifunctor-io from John was kind of right in my eyes. Once you do the work to be anything more specific, you might as well be explicit everywhere since one the flood gates open, that’s it.I ran into this issue when creating some specialized scala HTTP client’s that have a wide variety of algebras and the “HTTP” layer needed customizations so that it propa…

concurrent data pull

concurrent data pullSometimes you need to pull data concurrently from a database if the database is throttled or there are other parts of the system that constraining extract performance.For example, if you need to pull multiple partitions to upload to the cloud but the on-premise system is slow on the extract, concurrent extracts may be helpful. The below python script uses the multiprocessing module (vs threading) to handle multiple extracts at once based on a partition key you identify in the table to be extracted. The partition value should be a string or number, or anything that can be converted to a string that can be used in a filename. The script is not sophisticated and does not really support restart well but it does have some support for restart in case your extract is interrupted.Enhance as you see fit!#!/usr/bin/env pythonimport os import argparse import datetime from sqlalchemy import create_engine from multiprocessing import Pool import csvkit.utilities.sql2csv as c imp…