Ironic programmer: October 2011

Wednesday, October 19, 2011

Scala Stock Charts - part 6 some GUI things

I got a bit caught up in all the programming and finished up a few diagrams, then some refactoring and was able to remove a couple of hundred lines of code. The program now looks like this:

As you can see a lot have happened since the last pictures of the application. So here is some things.

Main part

The applications is build up around a main frame that contains the bar chart and some overlays. As you can see there is also some text fields and buttons. The upper part is to search stocks/indexes and also handle the overlays in the bar char. These fields and buttons are:

Search field to enter a search criteria to filter the instruments to search for.
Search button, to apply the search
A combo box, to show the filtered search result, and also choose the stock to view. This combo box listen to changes and updates the bar chart with the selected stock
Next there is a box to limit the range of the data. I have not introduced recalculation for splits and emissions so to have a possibility to handle to not show before these events, I added a box to filter away prices before this date part.
Next there is the intervals for the moving averages in the chart, default they are set to 5,20,50 but can be changed but have as most 3 different, but also as few as one. Default they are Simple Moving Average (SMA),
to change to Exponential Moving Average (EMA) there is a checkbox
to update changed settings for Moving averages there is a button "Update averages"
finally on the top there is a check box to switch on/off viewing of "Bollinger Bands"

In the bottom part there is buttons for additional chart types. these are the ones I picked as most interesting for me and they show some different aspect of the prices.

Stocastics (show how a stocks price is doing in relation to past movements)
RSI (shows how strong a stock is moving in the current direction)
Aroon (shows if a stock is trending or oscillating)
MACD (and also MACD-histogram) (a momentum oscillator)

As I mentioned the main part is a Bar chart diagram. It contains some overlays like volume, moving average and optional Bollinger bands. These help to get a basic overview of a stocks movement and trend. It is made up of a specialized panel for the bar chart to handle the drawing of the chart.

The main application implements a method named top that is used in wrapped Swing features used in Scala. I have an object representing my starting point of the application, the first lines looks like:
As you can see I provide a title and also here is the search field where to put a search string. The most important thing to note here is the inheritance of SimpleSwingApplication which makes this object a Swing application

The next part looks like this

/*
       * Search button, will get the pricelist and do filter and sort to present in combo box
       */
      val searchBtn = new Button("Search") {
          action = Action("Search"){ 
            val pricelists = PriceService.getPriceLists map { kv => kv.instruments } flatMap { 
                case i: Instrument => i 
                case x => x
              } filter {
                i: Instrument => i.name.toLowerCase().startsWith(searchField.text.toLowerCase)
              }
            
            val items = pricelists.toList sortWith {(i1,i2) => i1.name.toLowerCase < i2.name.toLowerCase}
            combo.peer.setModel(
              ComboBox.newConstantModel(
                  for{p <- items} yield p.ticker + ": " + p.name
              )
            )
          }
        }
        
      /*
       * ComboBox to present filtered search for stocks, see searchButton action code how search is performed
       */
      import event._
      val combo = new ComboBox(List("-- Search to be able to select --")) {
        preferredSize = new Dimension(300,20)
        listenTo(selection)
        reactions += {
          case SelectionChanged(c) => 
            curInstrument = new Instrument(selection.item.split(":")(0),selection.item.split(":")(1).trim,"")
            curPriceList = PriceService.getInstrumentData(curInstrument).filter(p => p.date.substring(0,fromTextBox.text.length).toInt >= fromTextBox.text.toInt)
            barchartPanel.updateList(curPriceList)
          case e => info(e.toString)
        }
      }
...
val chkBollinger = new CheckBox {
        text = "Bollinger bands"
        reactions += {
          case x => barchartPanel.setBollinger(this.selected) 
        }
      }

Here you can see some on the fly declaration and reactions to events on the components. The following code shows how to add these to the contents of other panels. you can also see the creation at the click of the button, creation of a generic diagram for "stochastics" mixed in with a DrawListener to make it drawable. Also how the series of stochastics is added to a generic diagram can be seen here.

/*
       * Top menu panel.
       */
      val menu = new FlowPanel {
        contents += new Label("Stock")
        contents += searchField
        contents += searchBtn
        contents += combo
        contents += fromTextBox
        contents += avgSettings
        contents += chkExpAverage
        contents += btnUpdateAvg
        contents += chkBollinger
      }
....
val bottomPanel = new FlowPanel{
        contents += new Button(){
          text = "Stochastics"
          reactions += {
            case bc: ButtonClicked => val stD = new Frame {
                title="Stochastics " + curInstrument.name
              }
              stD.open()
              val panel = new GenericDiagram with DrawListener
              val scrPan = new ScrollPane{
                preferredSize = new Dimension(1180,220)
                contents = panel
              }
              stD.contents = scrPan
              val stoList = PriceUtil.stochastics(curPriceList, 14)
              panel.addSerie(ChartSerie(stoList,Color.blue))
              panel.addSerie(ChartSerie(PriceUtil.slidingAvg(stoList, 3),Color.red))
              panel.addDateSerie(curPriceList.map{p=>p.date})
              panel.setLabelLevels(List(20d,50d,80d))
              // need to force set max, it is not always 100 ??
              panel.maxVal = 100d
          }
        }
....
contents = new BorderPanel() {
        add(menu, BorderPanel.Position.North) 
        add(scrollPane, BorderPanel.Position.Center)
        add(bottomPanel, BorderPanel.Position.South)
      }

This last part of the code adds the actual panels containing the fields, bar chart and buttons to the Swing application.

Additional panels

To add some extra check to the main part there is some diagrams that first was their own specialized panel, but after some refactoring they now all except MACD uses a generalized panel called GenericDiagram and here is a part of that code:

And here is addition of some of the buttons into menu panel (top panel) followed by some example of the bottom panel with addition of button and reactions to its event. You see here that here is where the panel frame is set up inside the reaction to show the GenericDiagram with "stochastic" data.
Not so much to mention really except that as you see I extend with a trait with basic chart properties. this is intended to have common properties for this and other diagrams. The var's is to be able to add series and date series to the diagram. So this panel is only responsible to draw the things that are sent to it.

Ability to draw

No bar chart is complete if it is not possible to draw some help lines. This is managed by adding a trait for this, that can be mixed in wherever I want it. The code for that trais is:

...
/**
 * Trait to use for Panels that should have line drawin possibilities
 */
trait DrawListener extends Panel {
  val pointZero = new Point(0,0)
  
  var curStart = pointZero//new Point(0,0)
  var curEnd = new Point(0,0)
  listenTo(mouse.clicks, mouse.moves, keys)
  reactions += {
    case e: MousePressed  => 
                curStart = e.point
         //moveTo(e.point)
         requestFocusInWindow()
       case e: MouseDragged  => curEnd = e.point; repaint() //lineTo(e.point)
       case e: MouseReleased => moveTo(curStart);lineTo(e.point);curStart=pointZero;curEnd=pointZero
       case KeyTyped(_,'c',_,_) => 
         path = new geom.GeneralPath
         repaint()
       case _ => repaint()
  }
  /* records the dragging */
  var path = new geom.GeneralPath

def lineTo(p: Point) { path.lineTo(p.x, p.y); repaint() }
  def moveTo(p: Point) { path.moveTo(p.x, p.y); repaint() }
  
  override def paintComponent(g: Graphics2D) = {
    super.paintComponent(g)
    g.setColor(Color.black)
    if(curStart.x != 0 && curEnd.y != 0){
      g.drawLine(curStart.x,curStart.y,curEnd.x,curEnd.y)
    }
    g.draw(path)
  }
}
...

An example of this functionality is to draw help lines in a MACD diagram. (or any really)

This can be mixed in like in this case:
And then it is possible to draw on that panel.

Saturday, October 1, 2011

Scala stocks part 5 - Refactoring, big time

There is soooo much stuff that I would like to look in to. Alfresco, Liferay, GWT, Spring, Scrum, Agile etc. Right now though, I continue with my Scala project doing a Technical analysis tool for stocks using a Swing gui, also that with Scalas Swing libraries.

After working with the GUI, I started to get annoyed on myself mixing things up here and there, so I decided to refactor a bit. Actually, since I wrote the last post, I got a bit further then just plain refactoring. Simultaneously with the refactoring I worked on the GUI and also restructured code to finish up the loading of prices, enhancement of that loading and in that work I also managed to introduce some "Actors".

I will now also start to explain a little bit more what the code does.

I will split the refactoring and explanation in more than this post, otherwise I may get too long.

Refactoring of the FileHandler.
The new file is now much shorter, and includes only stuff to handle files. The extraction and transformation to the model is removed from the file handling. that messy code can be seen here http://ironicprogrammer.blogspot.com/2011/08/scala-stock-charts-part-3-saving-quotes.html if you have not already read it.

FileHandler.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package me.ironic.scabors.file
import me.ironic.scabors.model._
import me.ironic.scabors.util.Logging

object FileHandler{
  def apply() = { new FileHandler() }
}

class FileHandler extends Logging {
  import java.io.{File,BufferedWriter,FileWriter,IOException}
  import scala.collection.mutable.{ HashMap => MutableHashMap}
  import scala.collection.immutable.{ HashMap }
  import scala.io.Source
  
  val rootDir = { scala.util.Properties.userDir }
  val filesDir = { rootDir + "/files"}
  val logFile = { rootDir + "/" + "sBors.log"}
  val pricelistsFile = { rootDir + "/files/pricelists.txt"}
      
  def checkDir(dirName: String){
     val f = new File(dirName)
     if(!f.exists){
       f.mkdirs
       info("!! Created dir at : " + dirName)
     }
  }
  
  def writeToFile(fileName: String, append: Boolean, rows: List[String]){
    // Check that dir exista or creat if not
    checkDir(filesDir)
    
    val bw = new BufferedWriter(new FileWriter(fileName,append))
    
    rows foreach { row =>
      bw.write(row + "\n")
    }
    
    bw.close()
  }
  
  def priceFile(ticker: String) = {
    rootDir + "/files/" + ticker.replace(" ","_") + ".txt"
  }
  
  def loadFile(fileName: String)  = {
    val fc = Source.fromFile(fileName)
    fc.getLines.toList
  }
  
}

In the start of the file, you see:
This is a companion object I use it to have a factory method. Maybe not really nescessary but at least I get away with not having to write
new FileHandler()
every time I need it and can use the shorter version
FileHandler()

The rest of the FileHandler should be quite easy to figure out. I have only one write-method where I send in the filename, if I want to append and a list of strings to write to the file. With this I can do all the saving I need.

There is also only one method for getting the file content (loadFile), this returns a list of strings which I later can traverse and to all the logic with.

I also have a method to have a format for how a price file is named. I do not want any blanks in the filename so this method is replacing that with "_" instead.

As you can see the FileHandler extends Logging. I do not need that here since no logging is going on in this class as it is now, but I can add it if I want. More about the Logging later.

Introducing PriceDAO.
To hide FileHandler from the rest of the system, and also to the mapping between files and model that was before done in the FileHandler, I introduced a DAO object to do the work. Doing this I can always change the way I persist later without too much impact on the code.

There is still some refactoring to do in this one since I have not changed the fact that I use mutable lists.

PriceDAO.scala

/*
 @author Peter Johansson / Ironic programmer 
 */
package me.ironic.scabors.dao

/**
 * Provides service methods and communicate with underlaying storage
 */
import me.ironic.scabors.util.Logging
import me.ironic.scabors.model._
import me.ironic.scabors.file.FileHandler
import scala.collection.mutable.{ HashMap => MutableHashMap}
import scala.collection.immutable.{ HashMap }
  
object PriceDAO extends Logging{
  
  def saveDailyPrices(prices: List[PriceItem]) = {
    info("## Saving daily prices")
       
    var lists = new MutableHashMap[String,List[Price]]
    var curList = "?"
    
    prices foreach { price =>
      price match { 
        case q: Price => { 
                            savePriceToFile(q);
                            lists(curList) = List(q) ::: lists(curList)
                          }
        case qList: PriceList => {
                                    curList = qList.name;
                                    lists += qList.name -> List()
                                 }
        case _ => warn(price.toString)
      }
    }
    
    // Save current lists to file...
    saveStockPriceList(lists)
  }
  
  /**
   * Method for saving a price to file
   */
  protected def savePriceToFile(price: Price) = {
    val fileHandler = FileHandler()
    val fName = fileHandler.priceFile(price.ticker)
    fileHandler.writeToFile(fName, true, List(price.toString))
  }
  
  /**
   * Method for saving an updated pricelist map to file.
   */
  protected def saveStockPriceList(pricelist: MutableHashMap[String,List[Price]]){
    info("## Saving pricelists")
    
    val sb = new StringBuffer
    sb.append("{ \"pricelists\": [\n")
    for{ (listname,prices) <- pricelist } {
      sb.append("\t{ \n\t\t\"listname\": \"" + listname + "\",\n")
      sb.append("\t\t\"stocks\": [\n")
      for{ price <- prices } {
        sb.append("\t\t\t{ \"ticker\": \""+price.ticker + "\", \"aktie\": \"" + price.aktie + "\"},\n")
      }
      sb.append("\t\t],\n")
      sb.append("\t},\n")
    }
    sb.append("}\n")
    
    val fileHandler = FileHandler()
    val fName = fileHandler.pricelistsFile
    fileHandler.writeToFile(fName, false, List(sb.toString))
  }
  
  /**
   * Method for loading price lists to a HashMap, 
   * each entry in the map is a list of Instruments
   */
  def loadPriceLists  : MutableHashMap[String, List[Instrument]] = {
    info("## Loading pricelists")
    var pricelists = new MutableHashMap[String, List[Instrument]]
    val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r
    val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r
   
    /*
     * When hitting a PriceList, a new list begins
     */
    var curList = "?"
    val fileHandler = FileHandler()
    val fName = fileHandler.pricelistsFile
    val rows = fileHandler.loadFile(fName)
    rows foreach { line =>
      line match {
        case ListExtractorRE(listname) => val pList = new PriceList(listname)
          curList = pList.name
          pricelists += pList.name -> List()
        case InstExtractorRE(ticker, aktie) => val inst = new Instrument(ticker, aktie) 
          pricelists(curList) = List(inst) ::: pricelists(curList)
        case _ => None
      }
      
    } 
    pricelists
  }
  
  /**
   * Method for loading all prices for a specific instrument.
   */
  def loadInstrumentData(inst: Instrument) : List[Price] = {
    val fileHandler = FileHandler()
    info("Loading instrument " + inst)
    val fName = fileHandler.priceFile(inst.ticker)
    
    val PriceExtractorRE = """Price\(([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*).*\)""".r
    val pricelines = fileHandler.loadFile(fName)
    val prices = pricelines map { line =>
      line match {
        case PriceExtractorRE(date,ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) => 
          Price(date,ticker,aktie,diffKr.toDouble,diffPerc.toDouble,buy.toDouble,sell.toDouble,last.toDouble,high.toDouble,low.toDouble,returnNo.toDouble,returnKr.toDouble)
        
      } 
    } 
    prices.toList
  }
  
}

Some explanation maybe?

Well, I am not going into every line. I choose some parts that I think is interesting parts of Scala.

This code it possibly a good candidate to do some further refactoring on to try to make it immutable, but this is what is going on.

lists is mutable, because I need to add things in it, and I assume that I do not know what lists are coming in. (Even if they are pretty known from the file)
prices is a list of type List[PriceItem], this means that everything in that list is of something under type PriceItem but right now I am only interested in the types Price and PriceList
If it is a PriceList I know that a new price list is starting, I add this as a map key with an empty list as the map value.
The flow goes on and for every Price that comes up this is added in the current list.

The next chunk of interest...

Here is the loading of the price lists. I will sow you part of the file so you know what we are dealing with.

That is how it is saved. So seeinging that makes it easier to explain what the above code does.

first there is 2 "extractors", these are regular expressions. In Scala you can use the 3-qouted strings to make a regular expression just by putting .r after it. This expressions can then be used as types cases in pattern matching and is very powerful way to extract thing.
(""".*"listname": "(.+)",.*\s*""".r) is for extracting the name of a list. This matches on the line in the file beginning with any character and then "listname":
the next "extractor" is for matching an instrument. for convenience I write the file with a "ticker" on a new line to avoid complex regexps. So I only need to parse the file and match on these.
When matching a new list I put a new map entry with an empty list, and when hitting a "ticker" I just add that instrument in the current list. Similar to what is going on when saving them.

Major restruction of PriceLoad.

I more or less totally reworked the loading of the prices from the site they are available from

One thing I wanted to do was to use actors. Here is were I got loose on doing that. Well, thats a bit exaggerating since there is only one Actor involved here (and another one in the Logging) so not so big fuzz. Here is the code, and I get into some details after it.

PriceLoad.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package me.ironic.scabors.dao
import me.ironic.scabors.model._
import me.ironic.scabors.util._
// Singleton to handle loading of Prices
object PriceFetcher extends Logging {
  import java.util.Calendar
  
  import java.text.SimpleDateFormat
    val fmt = new SimpleDateFormat("yyyyMMdd")
      
    def stringDateList(c: Calendar,bDate:String) : List[String] ={
      if(fmt.format(c.getTime()) == bDate) {
        List(fmt.format(c.getTime()))
      } else {
        val c2 = c.clone().asInstanceOf[Calendar]
        c2.add(Calendar.DATE,1)
        fmt.format(c.getTime()) :: stringDateList(c2,bDate)
      }
    }
      
   def main(args: Array[String]) = {
      if(args.length < 1  || args.length > 2){
        println("Usage: PriceFetcher YYYYMMDD # for single day")
        println("Usage: PriceFetcher YYYYMMDD YYYYMMDD # for batch mode")
        sys.exit(1)
      }
          
      val list = {
        if(args.length == 1){
          List(args(0))
        } else {
          require(args(1).compareTo(args(0)) > 0)
          val dFrom = fmt.parse(args(0))
          val cal = Calendar.getInstance()
          cal.setTime(dFrom)
          stringDateList(cal, args(1))
        }
      }
      
      info("Will run fetching for list " + list)
      val fetcher = new PriceFetchActor
      fetcher.start
      list foreach { s =>
        fetcher ! Daily(s)
      }
      fetcher ! Stop
      // Need to stop the logger, otherwise there is a hanging thread
      import scala.actors.Actor.State
      while(fetcher.getState != State.Terminated){
        // to wait until all work is done
      }
      stopLogger
      sys.exit(0)    
   }
}

import scala.actors.Actor
import scala.actors.Actor._
import me.ironic.scabors.util.Logging
class PriceFetchActor extends Actor with Logging{
  import scala.io.Source
      
  val baseUrl = "http://thesite/"
  def act(){
    loop{
      react {
        case d: Daily => 
          info("Fetching prices for " + d)
          val uri = baseUrl + d.date + "k.txt"
          try{
            val lines = readUrl(uri)
            val handledLines = processLines(lines,d.date)
            info("Did lines : " + handledLines.size)
            PriceDAO.saveDailyPrices(handledLines)
          } catch {
            case x => warn(x.toString)
          }
        case Stop => 
          info("Stopping PriceFetchActor")
          exit()
        case _ =>
      }
    }
  }
  
  def readUrl(url: String) : List[String] = {
    val fc = Source.fromURL(url).getLines().toList 
    fc
  }
  
  val PriceExtractorRE = """([[^Ticker]^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t.*""".r
  val ListExtractorRE = """(OMX[^\t]+)\t+""".r
  
  // Should go through the lines from the file and make a list with jSon-like format. 
  def processLines(lines: List[String],date: String) : List[PriceItem] = {
    info("-> Processing lines")
    val instruments = lines takeWhile(s=> ! s.startsWith("Externa listan")) 
    instruments map { inst =>  
       inst match {
        case ListExtractorRE(list) => PriceList(list)//"{\"instrumentList\": \"" + list + "\"}"
        case PriceExtractorRE(ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) 
          => Price(date,ticker,aktie,diffKr.replace(",",".").toDouble,diffPerc.replace(",",".").toDouble,buy.replace(",",".").toDouble,sell.replace(",",".").toDouble,last.replace(",",".").toDouble,high.replace(",",".").toDouble,low.replace(",",".").toDouble,returnNo.replace(",",".").toDouble,returnKr.replace(",",".").toDouble) 
        case _ => InvalidPrice(date,inst.replace("\t","[tab]"))
      } 
    }
  } 
}

This is an application object with a main method so it is available to run from the command line. What it does is to check if you enter with one or two arguments. Entering with one argument will load for one day. Entering with two arguments will run for a date span. I do not really car if there is days that does not have any prices, the only thing that will happen is that a FileNotFoundException will be logged.

The date span wil be recursively calculated to a list of strings with the method stringDateList.

After getting the list with dates, whether it is one item or more, I am traversing that list and sending a Daily object to an actor to do the loading and saving of that day. When all dates are handled I send a Stop message so the actor can exit itself.

Since I am sending to actors the flow of the program just passes to the end of the main method so I have to check the state of the actor before I do the exit. (There is a hang otherwise, because there is a thread that is still running somewhere.) and I do not want to do a sys.exit() before since that kills the thread that is loading.

The messages that is sent is case object/class that is defined like:

Other than the actor logic, the load and save is more or less the same as before. Just that now it is threaded and runs by an actor by sending messages to it.

Well, that was enough for this time. The GUI is more evolved also, but that is another story, another day. But a teaser to show how it looks so far: