Ironic programmer: Scala stocks part 5

There is soooo much stuff that I would like to look in to. Alfresco, Liferay, GWT, Spring, Scrum, Agile etc. Right now though, I continue with my Scala project doing a Technical analysis tool for stocks using a Swing gui, also that with Scalas Swing libraries.

After working with the GUI, I started to get annoyed on myself mixing things up here and there, so I decided to refactor a bit. Actually, since I wrote the last post, I got a bit further then just plain refactoring. Simultaneously with the refactoring I worked on the GUI and also restructured code to finish up the loading of prices, enhancement of that loading and in that work I also managed to introduce some "Actors".

I will now also start to explain a little bit more what the code does.

I will split the refactoring and explanation in more than this post, otherwise I may get too long.

Refactoring of the FileHandler.
The new file is now much shorter, and includes only stuff to handle files. The extraction and transformation to the model is removed from the file handling. that messy code can be seen here http://ironicprogrammer.blogspot.com/2011/08/scala-stock-charts-part-3-saving-quotes.html if you have not already read it.

FileHandler.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package me.ironic.scabors.file
import me.ironic.scabors.model._
import me.ironic.scabors.util.Logging

object FileHandler{
  def apply() = { new FileHandler() }
}

class FileHandler extends Logging {
  import java.io.{File,BufferedWriter,FileWriter,IOException}
  import scala.collection.mutable.{ HashMap => MutableHashMap}
  import scala.collection.immutable.{ HashMap }
  import scala.io.Source
  
  val rootDir = { scala.util.Properties.userDir }
  val filesDir = { rootDir + "/files"}
  val logFile = { rootDir + "/" + "sBors.log"}
  val pricelistsFile = { rootDir + "/files/pricelists.txt"}
      
  def checkDir(dirName: String){
     val f = new File(dirName)
     if(!f.exists){
       f.mkdirs
       info("!! Created dir at : " + dirName)
     }
  }
  
  def writeToFile(fileName: String, append: Boolean, rows: List[String]){
    // Check that dir exista or creat if not
    checkDir(filesDir)
    
    val bw = new BufferedWriter(new FileWriter(fileName,append))
    
    rows foreach { row =>
      bw.write(row + "\n")
    }
    
    bw.close()
  }
  
  def priceFile(ticker: String) = {
    rootDir + "/files/" + ticker.replace(" ","_") + ".txt"
  }
  
  def loadFile(fileName: String)  = {
    val fc = Source.fromFile(fileName)
    fc.getLines.toList
  }
  
}

In the start of the file, you see:
This is a companion object I use it to have a factory method. Maybe not really nescessary but at least I get away with not having to write
new FileHandler()
every time I need it and can use the shorter version
FileHandler()

The rest of the FileHandler should be quite easy to figure out. I have only one write-method where I send in the filename, if I want to append and a list of strings to write to the file. With this I can do all the saving I need.

There is also only one method for getting the file content (loadFile), this returns a list of strings which I later can traverse and to all the logic with.

I also have a method to have a format for how a price file is named. I do not want any blanks in the filename so this method is replacing that with "_" instead.

As you can see the FileHandler extends Logging. I do not need that here since no logging is going on in this class as it is now, but I can add it if I want. More about the Logging later.

Introducing PriceDAO.
To hide FileHandler from the rest of the system, and also to the mapping between files and model that was before done in the FileHandler, I introduced a DAO object to do the work. Doing this I can always change the way I persist later without too much impact on the code.

There is still some refactoring to do in this one since I have not changed the fact that I use mutable lists.

PriceDAO.scala

/*
 @author Peter Johansson / Ironic programmer 
 */
package me.ironic.scabors.dao

/**
 * Provides service methods and communicate with underlaying storage
 */
import me.ironic.scabors.util.Logging
import me.ironic.scabors.model._
import me.ironic.scabors.file.FileHandler
import scala.collection.mutable.{ HashMap => MutableHashMap}
import scala.collection.immutable.{ HashMap }
  
object PriceDAO extends Logging{
  
  def saveDailyPrices(prices: List[PriceItem]) = {
    info("## Saving daily prices")
       
    var lists = new MutableHashMap[String,List[Price]]
    var curList = "?"
    
    prices foreach { price =>
      price match { 
        case q: Price => { 
                            savePriceToFile(q);
                            lists(curList) = List(q) ::: lists(curList)
                          }
        case qList: PriceList => {
                                    curList = qList.name;
                                    lists += qList.name -> List()
                                 }
        case _ => warn(price.toString)
      }
    }
    
    // Save current lists to file...
    saveStockPriceList(lists)
  }
  
  /**
   * Method for saving a price to file
   */
  protected def savePriceToFile(price: Price) = {
    val fileHandler = FileHandler()
    val fName = fileHandler.priceFile(price.ticker)
    fileHandler.writeToFile(fName, true, List(price.toString))
  }
  
  /**
   * Method for saving an updated pricelist map to file.
   */
  protected def saveStockPriceList(pricelist: MutableHashMap[String,List[Price]]){
    info("## Saving pricelists")
    
    val sb = new StringBuffer
    sb.append("{ \"pricelists\": [\n")
    for{ (listname,prices) <- pricelist } {
      sb.append("\t{ \n\t\t\"listname\": \"" + listname + "\",\n")
      sb.append("\t\t\"stocks\": [\n")
      for{ price <- prices } {
        sb.append("\t\t\t{ \"ticker\": \""+price.ticker + "\", \"aktie\": \"" + price.aktie + "\"},\n")
      }
      sb.append("\t\t],\n")
      sb.append("\t},\n")
    }
    sb.append("}\n")
    
    val fileHandler = FileHandler()
    val fName = fileHandler.pricelistsFile
    fileHandler.writeToFile(fName, false, List(sb.toString))
  }
  
  /**
   * Method for loading price lists to a HashMap, 
   * each entry in the map is a list of Instruments
   */
  def loadPriceLists  : MutableHashMap[String, List[Instrument]] = {
    info("## Loading pricelists")
    var pricelists = new MutableHashMap[String, List[Instrument]]
    val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r
    val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r
   
    /*
     * When hitting a PriceList, a new list begins
     */
    var curList = "?"
    val fileHandler = FileHandler()
    val fName = fileHandler.pricelistsFile
    val rows = fileHandler.loadFile(fName)
    rows foreach { line =>
      line match {
        case ListExtractorRE(listname) => val pList = new PriceList(listname)
          curList = pList.name
          pricelists += pList.name -> List()
        case InstExtractorRE(ticker, aktie) => val inst = new Instrument(ticker, aktie) 
          pricelists(curList) = List(inst) ::: pricelists(curList)
        case _ => None
      }
      
    } 
    pricelists
  }
  
  /**
   * Method for loading all prices for a specific instrument.
   */
  def loadInstrumentData(inst: Instrument) : List[Price] = {
    val fileHandler = FileHandler()
    info("Loading instrument " + inst)
    val fName = fileHandler.priceFile(inst.ticker)
    
    val PriceExtractorRE = """Price\(([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*).*\)""".r
    val pricelines = fileHandler.loadFile(fName)
    val prices = pricelines map { line =>
      line match {
        case PriceExtractorRE(date,ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) => 
          Price(date,ticker,aktie,diffKr.toDouble,diffPerc.toDouble,buy.toDouble,sell.toDouble,last.toDouble,high.toDouble,low.toDouble,returnNo.toDouble,returnKr.toDouble)
        
      } 
    } 
    prices.toList
  }
  
}

Some explanation maybe?

Well, I am not going into every line. I choose some parts that I think is interesting parts of Scala.

This code it possibly a good candidate to do some further refactoring on to try to make it immutable, but this is what is going on.

lists is mutable, because I need to add things in it, and I assume that I do not know what lists are coming in. (Even if they are pretty known from the file)
prices is a list of type List[PriceItem], this means that everything in that list is of something under type PriceItem but right now I am only interested in the types Price and PriceList
If it is a PriceList I know that a new price list is starting, I add this as a map key with an empty list as the map value.
The flow goes on and for every Price that comes up this is added in the current list.

The next chunk of interest...

Here is the loading of the price lists. I will sow you part of the file so you know what we are dealing with.

That is how it is saved. So seeinging that makes it easier to explain what the above code does.

first there is 2 "extractors", these are regular expressions. In Scala you can use the 3-qouted strings to make a regular expression just by putting .r after it. This expressions can then be used as types cases in pattern matching and is very powerful way to extract thing.
(""".*"listname": "(.+)",.*\s*""".r) is for extracting the name of a list. This matches on the line in the file beginning with any character and then "listname":
the next "extractor" is for matching an instrument. for convenience I write the file with a "ticker" on a new line to avoid complex regexps. So I only need to parse the file and match on these.
When matching a new list I put a new map entry with an empty list, and when hitting a "ticker" I just add that instrument in the current list. Similar to what is going on when saving them.

Major restruction of PriceLoad.

I more or less totally reworked the loading of the prices from the site they are available from

One thing I wanted to do was to use actors. Here is were I got loose on doing that. Well, thats a bit exaggerating since there is only one Actor involved here (and another one in the Logging) so not so big fuzz. Here is the code, and I get into some details after it.

PriceLoad.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package me.ironic.scabors.dao
import me.ironic.scabors.model._
import me.ironic.scabors.util._
// Singleton to handle loading of Prices
object PriceFetcher extends Logging {
  import java.util.Calendar
  
  import java.text.SimpleDateFormat
    val fmt = new SimpleDateFormat("yyyyMMdd")
      
    def stringDateList(c: Calendar,bDate:String) : List[String] ={
      if(fmt.format(c.getTime()) == bDate) {
        List(fmt.format(c.getTime()))
      } else {
        val c2 = c.clone().asInstanceOf[Calendar]
        c2.add(Calendar.DATE,1)
        fmt.format(c.getTime()) :: stringDateList(c2,bDate)
      }
    }
      
   def main(args: Array[String]) = {
      if(args.length < 1  || args.length > 2){
        println("Usage: PriceFetcher YYYYMMDD # for single day")
        println("Usage: PriceFetcher YYYYMMDD YYYYMMDD # for batch mode")
        sys.exit(1)
      }
          
      val list = {
        if(args.length == 1){
          List(args(0))
        } else {
          require(args(1).compareTo(args(0)) > 0)
          val dFrom = fmt.parse(args(0))
          val cal = Calendar.getInstance()
          cal.setTime(dFrom)
          stringDateList(cal, args(1))
        }
      }
      
      info("Will run fetching for list " + list)
      val fetcher = new PriceFetchActor
      fetcher.start
      list foreach { s =>
        fetcher ! Daily(s)
      }
      fetcher ! Stop
      // Need to stop the logger, otherwise there is a hanging thread
      import scala.actors.Actor.State
      while(fetcher.getState != State.Terminated){
        // to wait until all work is done
      }
      stopLogger
      sys.exit(0)    
   }
}

import scala.actors.Actor
import scala.actors.Actor._
import me.ironic.scabors.util.Logging
class PriceFetchActor extends Actor with Logging{
  import scala.io.Source
      
  val baseUrl = "http://thesite/"
  def act(){
    loop{
      react {
        case d: Daily => 
          info("Fetching prices for " + d)
          val uri = baseUrl + d.date + "k.txt"
          try{
            val lines = readUrl(uri)
            val handledLines = processLines(lines,d.date)
            info("Did lines : " + handledLines.size)
            PriceDAO.saveDailyPrices(handledLines)
          } catch {
            case x => warn(x.toString)
          }
        case Stop => 
          info("Stopping PriceFetchActor")
          exit()
        case _ =>
      }
    }
  }
  
  def readUrl(url: String) : List[String] = {
    val fc = Source.fromURL(url).getLines().toList 
    fc
  }
  
  val PriceExtractorRE = """([[^Ticker]^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t.*""".r
  val ListExtractorRE = """(OMX[^\t]+)\t+""".r
  
  // Should go through the lines from the file and make a list with jSon-like format. 
  def processLines(lines: List[String],date: String) : List[PriceItem] = {
    info("-> Processing lines")
    val instruments = lines takeWhile(s=> ! s.startsWith("Externa listan")) 
    instruments map { inst =>  
       inst match {
        case ListExtractorRE(list) => PriceList(list)//"{\"instrumentList\": \"" + list + "\"}"
        case PriceExtractorRE(ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) 
          => Price(date,ticker,aktie,diffKr.replace(",",".").toDouble,diffPerc.replace(",",".").toDouble,buy.replace(",",".").toDouble,sell.replace(",",".").toDouble,last.replace(",",".").toDouble,high.replace(",",".").toDouble,low.replace(",",".").toDouble,returnNo.replace(",",".").toDouble,returnKr.replace(",",".").toDouble) 
        case _ => InvalidPrice(date,inst.replace("\t","[tab]"))
      } 
    }
  } 
}

This is an application object with a main method so it is available to run from the command line. What it does is to check if you enter with one or two arguments. Entering with one argument will load for one day. Entering with two arguments will run for a date span. I do not really car if there is days that does not have any prices, the only thing that will happen is that a FileNotFoundException will be logged.

The date span wil be recursively calculated to a list of strings with the method stringDateList.

After getting the list with dates, whether it is one item or more, I am traversing that list and sending a Daily object to an actor to do the loading and saving of that day. When all dates are handled I send a Stop message so the actor can exit itself.

Since I am sending to actors the flow of the program just passes to the end of the main method so I have to check the state of the actor before I do the exit. (There is a hang otherwise, because there is a thread that is still running somewhere.) and I do not want to do a sys.exit() before since that kills the thread that is loading.

The messages that is sent is case object/class that is defined like:

Other than the actor logic, the load and save is more or less the same as before. Just that now it is threaded and runs by an actor by sending messages to it.

Well, that was enough for this time. The GUI is more evolved also, but that is another story, another day. But a teaser to show how it looks so far:

Ironic programmer

Saturday, October 1, 2011

Scala stocks part 5 - Refactoring, big time

No comments:

Post a Comment

Visitors