Ironic programmer: August 2011

Monday, August 22, 2011

Scala stock charts, part 3 - Saving the quotes

Going further in programming the functionality behind the technical analysis tool for stocks made in Scala we are going to save the daily quotes to separate file, we are also going to supply functionality to load the data for a stock, and save and read a price list divided in 3 lists.

The format of the files is a bitt different, the Quotes I will save and read as they are, but the price list will have a different format that will look as jSon and is saved in a way that will be easy to read with pattern matching. For the matter of fact also the Stock-quotes for a single instrument will be read and transformed to a list of Quote's with pattern matching.

FileHandler.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package scala.ironic.file

object FileHandler {
  import java.io.{File,BufferedWriter,FileWriter,IOException}
  import scala.collection.mutable.{ HashMap => MutableHashMap}
  import scala.collection.immutable.{ HashMap }
  import scala.io.Source
  
  val rootDir = { scala.util.Properties.userDir }
  val filesDir = { rootDir + "/files"}
  val logFile = { rootDir + "/" + "sBors.log"}
  val pricelistsFile = { rootDir + "/files/pricelists.txt"}
    
  def saveDailyQuotes(quotes: Iterator[QuoteItem]) = {
    log("## Saving daily qoutes")
    log("--- rootDir : " + rootDir)
    log("--- filesDir : " + filesDir)
    // Check that dir exista or creat if not
    checkDir(filesDir)
    
    var lists = new MutableHashMap[String,List[Quote]]
    var curList = "?"
    
    quotes foreach { quote =>
      quote match { 
        case q: Quote => { 
                            saveQuoteToFile(q);
                            lists(curList) = List(q) ::: lists(curList)
                          }
        case qList: QuoteList => {
                                    curList = qList.name;
                                    lists += qList.name -> List()
                                 }
        case _ => log("! " + quote)
      }
    }
    
    // Save current lists to file...
    saveStockPriceList(lists)
    
  }
  
  def checkDir(dirName: String){
     val f = new File(dirName)
     if(!f.exists){
       f.mkdirs
       log("!! Created dir at : " + dirName)
     }
  }
  
  def saveQuoteToFile(quote: Quote) = {
    val fileName = scala.util.Properties.userDir + "/files/" + quote.ticker.replace(" ","_") + ".txt"
    val bw = new BufferedWriter(new FileWriter(fileName,true))
    bw.write(quote + "\n")
    bw.close()
  }
  
  def saveStockPriceList(pricelist: MutableHashMap[String,List[Quote]]){
    log("## Saving pricelists")
    val bw = new BufferedWriter(new FileWriter(pricelistsFile,false))
    val sb = new StringBuffer
    sb.append("{ \"pricelists\": [\n")
    for{ (listname,quotes) <- pricelist } {
      sb.append("\t{ \n\t\t\"listname\": \"" + listname + "\",\n")
      sb.append("\t\t\"stocks\": [\n")
      for{ quote <- quotes } {
        sb.append("\t\t\t{ \"ticker\": \""+quote.ticker + "\", \"aktie\": \"" + quote.aktie + "\"},\n")
      }
      sb.append("\t\t],\n")
      sb.append("\t},\n")
    }
    sb.append("}\n")
    
    bw.write(sb.toString)
    bw.close()
  }
  
  def log(msg: String){
    println(msg)
    val bw = new BufferedWriter(new FileWriter(logFile,true))
    bw.write(msg + "\n")
    bw.close()
    
  }
  
  def loadPriceLists : HashMap[String, List[Instrument]] = {
    log("## Loading pricelists")
    val pricelists = new HashMap[String, List[Instrument]]
    val fc = Source.fromFile(pricelistsFile)
    val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r
    val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r
   
    fc.getLines foreach { line =>
      line match {
        case ListExtractorRE(listname) => println("#### got listname : " + new QuoteList(listname))
        case InstExtractorRE(ticker, aktie) => 
          println("# got instrument : " + new Instrument(ticker, aktie))
        case _ => 
      }
      
    }
    return pricelists
  }
  
  def loadInstrument(ticker: String): List[QuoteItem] = {
    val fn = rootDir + "/files/" + ticker.replace(" ","_") + ".txt"
    val fc = Source.fromFile(fn)
    val QuoteExtractorRE = """Quote\(([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*).*\)""".r
    println(QuoteExtractorRE)
    val quotelines = fc.getLines// filter( s=> ! s.startsWith("Quote")) 
    val quotes = quotelines map { line =>
      line match {
        case QuoteExtractorRE(date,ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) => 
          Quote(date,ticker,aktie,diffKr.toDouble,diffPerc.toDouble,buy.toDouble,sell.toDouble,last.toDouble,high.toDouble,low.toDouble,returnNo.toDouble,returnKr.toDouble)
        case _ => InvalidQuote("?", line)
      } 
    } 
    return quotes.toList
  }
  
  
}

That is a lot of code that I wrote on the fly, but not as much as it would have been to do the same thing in Java. the nice thing is the pattern matching, that saves a lot pain that Java code would bring, for example reading every line in the file would not be more code, but also creating and getting the values to objects would give a lot of more code.

[coming... Java code to do it as a comparison]

checkDir: is a method to create directory structure if it does not exist. Actually the first time loading the quotes there is no /files dir if it is not created manually, to avoid crashing, and open upp for the possibility to later use some config to have the data files in a directory of choice this is to ensure that the supplied configured dir will be created.

saveDailyQuotes: traverse the content of a daily quote file and calls to save single quote and collect the price lists and calls to save these.

saveStockPriceList: saves the current instruments divided in their lists to a file.

saveQuoteToFile: append a single quote to its data file.

Finally the methods for loading price lists and a single instrument

loadPriceLists: loads the pice lists back to a map with a list name and a list of Instruments. (Actually it is note yet finished, since I want to think out a way to avoid having a mutable HashMap, now it only prints out what it gets.)

loadInstrument: loads the file of a single stock to a list of Quote's

I am not completely happy with it, there is some duplication of code, for example I create new Writers in every method that save things. There is certainly things that can improve here. However, refactoring is a later issue, and this does what I want it to do, so for now it will have to do.

Now the interesting part is beginning to appear, next thing is to make functionality to take car of the technical stuff for the charts to be drawn.

Friday, August 19, 2011

Scala stock charts, part 2 - Getting the quotes

I am Swedish so I am for the moment only interested in the Swedish stock market. I have found a site where to get the quotes, it is in text format and I intend to read this information and save it in in other files, one file for each instrument (I call on single stock instrument since it is a form of financial instrument).

The format of the daily file is: (There is an example file to work with at : http://sites.google.com/site/ironicprogrammer/files/20110817k.txt)

Slutkurser från OMX 2011-08-17 Slutkurser från Burgundy 2011-08-17
Ticker Aktie +/- +/-% Köp Sälj Senast Högst Lägst Oms (antal) Oms (SEK) Senast Högst Lägst Oms (antal) Oms (SEK)
OMX Stockholm Large Cap
ABB ABB Ltd -2,30 -1,66 136,40 136,50 136,40 139,40 136,40 3299500 453921900 136,6 139,4 136,6 183000 25240000

It is hard to see in that little box, however. first row is title information of the file.
Slutkurser från OMX 2011-08-17[6 tabs]Slutkurser från Burgundy 2011-08-17

That is not really necessary and can be skipped as the next line, but the next line is interesting to see the structure of the quotes:

Ticker Aktie +/- +/-% Köp Sälj Senast Högst Lägst Oms (antal) Oms (SEK)[6 tabs]Senast Högst Lägst Oms (antal) Oms (SEK)

When reading the file I will only be interested in the quotes from OMX, the Burgundy I do not want so when saving the quotes to different files I will not consider these, so it is the blue marked fileds that would be used.

The third row

OMX Stockholm Large Cap

is the name of the first list of quotes. It continues like this in the file;
Listname
Ticker Aktie ...
...
Listname ...

Ticker Aktie ...
...

I am only interested in the first three lists, the forth is named "Externa listan", and I will use this name to break the read.

So lets start write som code (finally!!)

I use the REPL for a little bit trial before I put in in classes/objects.

To test to read the file:

Ok, everything seams to work so this need to be taken car of in some other way.

Lets put it in a bit more organized way:

I put a file in src/main/scala/ironic that is named QuoteLoad.scala with the following content:

QuoteLoad.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package scala.ironic

// To be able to handle Iterator[QuoteItem]
trait QuoteItem 

case class InvalidQuote(date: String, content:String) extends QuoteItem
case class QuoteList(name:String) extends QuoteItem
case class Quote(date: String, ticker :String, aktie : String,
    diffKr : Double, diffPerc : Double, buy : Double,
    sell : Double, last : Double, high : Double,
    low : Double, returnNo : Double, returnKr : Double
  ) extends QuoteItem

// Singleton to handle loading of quotes
object QuoteLoad {
   def main(args: Array[String]) = {
      
      loadQuotes("http://sites.google.com/site/ironicprogrammer/files/","20110817")
   }
   
   val QuoteExtractorRE = """([[^Ticker]^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t.*""".r
   val ListExtractorRE = """(OMX[^\t]+)\t+""".r
  
   def loadQuotes(url: String, date: String){
      import scala.io.Source
      
      val uri = url + date + "k.txt"
      val fc = Source.fromURL(uri) 

      val instruments = processLines(fc.getLines(),date)
      println (instruments)
      instruments foreach { inst => inst match {
        case ilist: QuoteList => 
            println(ilist)
        case q: Quote =>
            println(q)
        case _ => println(inst)//
        }
      }
   }
  
    // Should go through the lines from the file and make a iterable with QuoteItem's. 
    def processLines(lines: Iterator[String],date: String) : Iterator[QuoteItem] = {
      val instruments = lines takeWhile(s=> ! s.startsWith("Externa listan")) 
      instruments map { inst =>  
         inst match {
          case ListExtractorRE(list) => QuoteList(list)//"{\"instrumentList\": \"" + list + "\"}"
   case QuoteExtractorRE(ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) 
            => Quote(date,ticker,aktie,diffKr.replace(",",".").toDouble,diffPerc.replace(",",".").toDouble,buy.replace(",",".").toDouble,sell.replace(",",".").toDouble,last.replace(",",".").toDouble,high.replace(",",".").toDouble,low.replace(",",".").toDouble,returnNo.replace(",",".").toDouble,returnKr.replace(",",".").toDouble) 
          case _ => InvalidQuote(date,inst.replace("\t","[tab]"))
        } 
      } 
    }
    
}

Ok there may be some features to explain here.
First of all, I make use of an singleton object here so I will be able to call from a terminal window. So it has a main method, but I also extract the functionality so the future client will be able to use the same methods to load quotes.

There is a trait so I can avoid getting Iterator[Any] and instead getting Iterator[QuoteItem] to be able to handle the case class objects only, when I later will save the quotes to different files.

QuoteList: is for holding the name of a list (there is different list of quotes according to its category, that is more or less the amount of trading that is done and the size of the companies)
Quote: is for holding the quote for the day for a specific share
InvalidQuote: is for holding (and logging for error checking) an invalid row from the file.

There is two regexp's that is handled later in pattern matching to extract the lists and quotes from the file. They look for the structure in the file for valid quotes and the simpler one is for extracting the list name that is only one text.

def loadQuotes(url: String, date: String): handles the load from the URL and calls a helper method to extract the valid quotes and lists.

This will have to do for a first version, later I will save these quotes to different files.