Scala stock charts, part 3 - Saving the quotes

Going further in programming the functionality behind the technical analysis tool for stocks made in Scala we are going to save the daily quotes to separate file, we are also going to supply functionality to load the data for a stock, and save and read a price list divided in 3 lists.

The format of the files is a bitt different, the Quotes I will save and read as they are, but the price list will have a different format that will look as jSon and is saved in a way that will be easy to read with pattern matching. For the matter of fact also the Stock-quotes for a single instrument will be read and transformed to a list of Quote's with pattern matching.

FileHandler.scala

/* @author Peter Johansson / Ironic programmer */ package scala.ironic.file object FileHandler { import java.io.{File,BufferedWriter,FileWriter,IOException} import scala.collection.mutable.{ HashMap => MutableHashMap} import scala.collection.immutable.{ HashMap } import scala.io.Source val rootDir = { scala.util.Properties.userDir } val filesDir = { rootDir + "/files"} val logFile = { rootDir + "/" + "sBors.log"} val pricelistsFile = { rootDir + "/files/pricelists.txt"} def saveDailyQuotes(quotes: Iterator[QuoteItem]) = { log("## Saving daily qoutes") log("--- rootDir : " + rootDir) log("--- filesDir : " + filesDir) // Check that dir exista or creat if not checkDir(filesDir) var lists = new MutableHashMap[String,List[Quote]] var curList = "?" quotes foreach { quote => quote match { case q: Quote => { saveQuoteToFile(q); lists(curList) = List(q) ::: lists(curList) } case qList: QuoteList => { curList = qList.name; lists += qList.name -> List() } case _ => log("! " + quote) } } // Save current lists to file... saveStockPriceList(lists) } def checkDir(dirName: String){ val f = new File(dirName) if(!f.exists){ f.mkdirs log("!! Created dir at : " + dirName) } } def saveQuoteToFile(quote: Quote) = { val fileName = scala.util.Properties.userDir + "/files/" + quote.ticker.replace(" ","_") + ".txt" val bw = new BufferedWriter(new FileWriter(fileName,true)) bw.write(quote + "\n") bw.close() } def saveStockPriceList(pricelist: MutableHashMap[String,List[Quote]]){ log("## Saving pricelists") val bw = new BufferedWriter(new FileWriter(pricelistsFile,false)) val sb = new StringBuffer sb.append("{ \"pricelists\": [\n") for{ (listname,quotes) <- pricelist } { sb.append("\t{ \n\t\t\"listname\": \"" + listname + "\",\n") sb.append("\t\t\"stocks\": [\n") for{ quote <- quotes } { sb.append("\t\t\t{ \"ticker\": \""+quote.ticker + "\", \"aktie\": \"" + quote.aktie + "\"},\n") } sb.append("\t\t],\n") sb.append("\t},\n") } sb.append("}\n") bw.write(sb.toString) bw.close() } def log(msg: String){ println(msg) val bw = new BufferedWriter(new FileWriter(logFile,true)) bw.write(msg + "\n") bw.close() } def loadPriceLists : HashMap[String, List[Instrument]] = { log("## Loading pricelists") val pricelists = new HashMap[String, List[Instrument]] val fc = Source.fromFile(pricelistsFile) val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r fc.getLines foreach { line => line match { case ListExtractorRE(listname) => println("#### got listname : " + new QuoteList(listname)) case InstExtractorRE(ticker, aktie) => println("# got instrument : " + new Instrument(ticker, aktie)) case _ => } } return pricelists } def loadInstrument(ticker: String): List[QuoteItem] = { val fn = rootDir + "/files/" + ticker.replace(" ","_") + ".txt" val fc = Source.fromFile(fn) val QuoteExtractorRE = """Quote\(([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*).*\)""".r println(QuoteExtractorRE) val quotelines = fc.getLines// filter( s=> ! s.startsWith("Quote")) val quotes = quotelines map { line => line match { case QuoteExtractorRE(date,ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) => Quote(date,ticker,aktie,diffKr.toDouble,diffPerc.toDouble,buy.toDouble,sell.toDouble,last.toDouble,high.toDouble,low.toDouble,returnNo.toDouble,returnKr.toDouble) case _ => InvalidQuote("?", line) } } return quotes.toList } } That is a lot of code that I wrote on the fly, but not as much as it would have been to do the same thing in Java. the nice thing is the pattern matching, that saves a lot pain that Java code would bring, for example reading every line in the file would not be more code, but also creating and getting the values to objects would give a lot of more code.

[coming... Java code to do it as a comparison]

checkDir: is a method to create directory structure if it does not exist. Actually the first time loading the quotes there is no /files dir if it is not created manually, to avoid crashing, and open upp for the possibility to later use some config to have the data files in a directory of choice this is to ensure that the supplied configured dir will be created.

saveDailyQuotes: traverse the content of a daily quote file and calls to save single quote and collect the price lists and calls to save these.

saveStockPriceList: saves the current instruments divided in their lists to a file.

saveQuoteToFile: append a single quote to its data file.

Finally the methods for loading price lists and a single instrument

loadPriceLists: loads the pice lists back to a map with a list name and a list of Instruments. (Actually it is note yet finished, since I want to think out a way to avoid having a mutable HashMap, now it only prints out what it gets.)

loadInstrument: loads the file of a single stock to a list of Quote's


I am not completely happy with it, there is some duplication of code, for example I create new Writers in every method that save things. There is certainly things that can improve here. However, refactoring is a later issue, and this does what I want it to do, so for now it will have to do.

Now the interesting part is beginning to appear, next thing is to make functionality to take car of the technical stuff for the charts to be drawn.

Scala stock charts, part 2 - Getting the quotes

I am Swedish so I am for the moment only interested in the Swedish stock market. I have found a site where to get the quotes, it is in text format and I intend to read this information and save it in in other files, one file for each instrument (I call on single stock instrument since it is a form of financial instrument).

The format of the daily file is: (There is an example file to work with at : http://sites.google.com/site/ironicprogrammer/files/20110817k.txt)

Slutkurser från OMX 2011-08-17 Slutkurser från Burgundy 2011-08-17
Ticker Aktie +/- +/-% Köp Sälj Senast Högst Lägst Oms (antal) Oms (SEK) Senast Högst Lägst Oms (antal) Oms (SEK)
OMX Stockholm Large Cap
ABB ABB Ltd -2,30 -1,66 136,40 136,50 136,40 139,40 136,40 3299500 453921900 136,6 139,4 136,6 183000 25240000

It is hard to see in that little box, however. first row is title information of the file.
Slutkurser från OMX 2011-08-17[6 tabs]Slutkurser från Burgundy 2011-08-17

That is not really necessary and can be skipped as the next line, but the next line is interesting to see the structure of the quotes:

Ticker Aktie +/- +/-% Köp Sälj Senast Högst Lägst Oms (antal) Oms (SEK)[6 tabs]Senast Högst Lägst Oms (antal) Oms (SEK)

 When reading the file I will only be interested in the quotes from OMX, the Burgundy I do not want so when saving the quotes to different files I will not consider these, so it is the blue marked fileds that would be used.

The third row

OMX Stockholm Large Cap

is the name of the first list of quotes. It continues like this in the file;
Listname
Ticker Aktie ...
...
Listname ...

Ticker Aktie ...
...

I am only interested in the first three lists, the forth is named "Externa listan", and I will use this name to break the read.

So lets start write som code (finally!!)

I use the REPL for a little bit trial before I put in in classes/objects.

To test to read the file:

scala> val url = "http://sites.google.com/site/ironicprogrammer/files/" url: java.lang.String = http://sites.google.com/site/ironicprogrammer/files/ scala> val file = "20110817k.txt" file: java.lang.String = 20110817k.txt scala> import scala.io.Source import scala.io.Source scala> val fc = Source.fromURL(url+file) fc: scala.io.BufferedSource = non-empty iterator scala>; fc.getLines foreach { println } Slutkurser fr?n OMX 2011-08-17 Slutkurser fr?n Burgundy 2011-08-17 Ticker Aktie +/- +/-% K?p S?lj Senast H?gst L?gst Oms (antal) Oms (SEK) Senast H?gst L?gsOms (antal) Oms (SEK)<br /> OMX Stockholm Large Cap ABB ABB Ltd -2,30 -1,66 136,40 136,50 136,40 139,40 136,40 3299500 453921900 136,6 139,4 136,6 183000 25240000<br /> ALFA Alfa Laval -0,80 -0,68 117,50 117,60 117,50 119,10 114,80 1121500 131304100 117,6 118,9 114,49223 5760000

Ok, everything seams to work so this need to be taken car of in some other way.

Lets put it in a bit more organized way:

I put a file in src/main/scala/ironic that is named QuoteLoad.scala with the following content:

QuoteLoad.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package scala.ironic

// To be able to handle Iterator[QuoteItem]
trait QuoteItem 

case class InvalidQuote(date: String, content:String) extends QuoteItem
case class QuoteList(name:String) extends QuoteItem
case class Quote(date: String, ticker :String, aktie : String,
    diffKr : Double, diffPerc : Double, buy : Double,
    sell : Double, last : Double, high : Double,
    low : Double, returnNo : Double, returnKr : Double
  ) extends QuoteItem

// Singleton to handle loading of quotes
object QuoteLoad {
   def main(args: Array[String]) = {
      
      loadQuotes("http://sites.google.com/site/ironicprogrammer/files/","20110817")
   }
   
   val QuoteExtractorRE = """([[^Ticker]^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t.*""".r
   val ListExtractorRE = """(OMX[^\t]+)\t+""".r
  
   def loadQuotes(url: String, date: String){
      import scala.io.Source
      
      val uri = url + date + "k.txt"
      val fc = Source.fromURL(uri) 

      val instruments = processLines(fc.getLines(),date)
      println (instruments)
      instruments foreach { inst => inst match {
        case ilist: QuoteList => 
            println(ilist)
        case q: Quote =>
            println(q)
        case _ => println(inst)//
        }
      }
   }
  
    // Should go through the lines from the file and make a iterable with QuoteItem's. 
    def processLines(lines: Iterator[String],date: String) : Iterator[QuoteItem] = {
      val instruments = lines takeWhile(s=> ! s.startsWith("Externa listan")) 
      instruments map { inst =>  
         inst match {
          case ListExtractorRE(list) => QuoteList(list)//"{\"instrumentList\": \"" + list + "\"}"
   case QuoteExtractorRE(ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) 
            => Quote(date,ticker,aktie,diffKr.replace(",",".").toDouble,diffPerc.replace(",",".").toDouble,buy.replace(",",".").toDouble,sell.replace(",",".").toDouble,last.replace(",",".").toDouble,high.replace(",",".").toDouble,low.replace(",",".").toDouble,returnNo.replace(",",".").toDouble,returnKr.replace(",",".").toDouble) 
          case _ => InvalidQuote(date,inst.replace("\t","[tab]"))
        } 
      } 
    }
    
}

Ok there may be some features to explain here. 
First of all, I make use of an singleton object here so I will be able to call from a terminal window. So it has a main method, but I also extract the functionality so the future client will be able to use the same methods to load quotes.

There is a trait so I can avoid getting Iterator[Any] and instead getting Iterator[QuoteItem] to be able to handle the case class objects only, when I later will save the quotes to different files.

QuoteList: is for holding the name of a list (there is different list of quotes according to its category, that is more or less the amount of trading that is done and the size of the companies)
Quote: is for holding the quote for the day for a specific share
InvalidQuote: is for holding (and logging for error checking) an invalid row from the file.

There is two regexp's that is handled later in pattern matching to extract the lists and quotes from the file. They look for the structure in the file for valid quotes and the simpler one is for extracting the list name that is only one text.

def loadQuotes(url: String, date: String): handles the load from the URL and calls a helper method to extract the valid quotes and lists.

This will have to do for a first version, later I will save these quotes to different files.

Scala stock charts, part 1 - Thoughts about structure

Having read through almost halfway, and not even coming to the functional part, in the online book "Programming Scala" by Alex Payne (http://ofps.oreilly.com/titles/9780596155957), I am starting to get some ideas of what I want to try out in my new Technical Analysis program for stocks.  For sure I would like to use SBT (https://github.com/harrah/xsbt/wiki) as a building tool and try to use "Specs" for tests (http://ofps.oreilly.com/titles/9780596155957/ScalaToolsLibs.html#ScalaSpecs).

To start with I want to build my program using either common classes or case classes, haven´t clearly figured that out, at least I want to use companion objects. Anyway, I want to get in touch with different features in Scala so there probably will be some file handling with regexp in Scalas pattern matching. At the same way as I want to get in touch with as many features as possible, I also want to keep it as simple as possible so there probably will be som rewrites during the course of my little project as I learn more. If possible I will use trait also, but probably after a rework. I will start more "java-like" with more OO-structure and then try to "functionalize" it.

I haven´t really figured out if I want o use JavaFX either. So I will try to build a library in Scala that ccan be used by client of choice, may be JavaFX but maybe also Flex or Swing. That I will figure out later.

More about this later....
and then probably with some code :)

Long time no see

It has been a while since last post. Sorry about that! It has been a period of organizational problems (not as fun as programming problems that needs to be solved) that could have been written about, but to sit in the middle of a stormy change takes the energy away.

Anyway, this is what I have been doing:
I started looking at Scala. For me it looks really interesting and I think it is something that is worth looking in to since it joins well with Java and runs on the JVM. You also need to write less code to do the same thing as with Java.
To learn Scala I have small idea that I should take my old Java-Swing tool for technical analysis of stocks on the Swedish stock market, set it up to use Scala with a JavaFX-client for the GUI. In this way I intend to get into Scala and at the same time JavaFX a little bit. this little tool of mine I have had as a project to learn new programming languages for over 10 years now. From the first version in an Excel spreadsheet it has taken the way over Turbo-pascal, Visual C++, maybe I also had a C#-client for a while until I returned to the Java Swing client.

I also started to read a lot of agile books about modelling and developing and also a bit about Neuro Linguistic Programming but that is a completely different subject but may be good if you want to take a leading role and/or how to handle the users and other team members and just get better at asking questions as one area of self development.

I will go on, so maybe there will soon be a posting about Scala or JavaFX or both. Or something completely different.

Ciao

Problem with Grails on Glassfish v3?

I almost gave up trying to run grails applications on Glassfish v3. Today I stumbled on a solution that was so easy that I should be ashamed I did not investigate a bit further. The problem is with some OSGi headers and is documented here.

Others have had the problem and the solution I found on nabble was pretty simple:
Add the following line in BuildConfig.groovy in your project that should run on a Glassfish v3.

grails.project.war.osgi.headers=false

This helps with the problem not being able to deploy a grails app on Glassfish v3 when you experience the following hints in the log on GF:

SEVERE: Module type not recognized
SEVERE: There is no installed container capable of handling this application com.sun.enterprise.deploy.shared.FileArchive@68237947

Mountain climbing as analogy to programming.

I just had vacation and I do not know why, but suddenly I got to think that walking/climbing a mountain (as that is one of my favorite things to do at vacation) could be used as an analogy for programming and learning programming.

As a starting point we have a geek that have no physical ability to talk about (because geeks seldom do sports or go outside the house).

Start with easy tracks

To be able to climb a mountain you need to be fit. If you start with a very steep path without being fit, you will get exhausted or fall down and die.

Also keep the distance short so you do not overuse your muscles. you may want to take a hike to morrow as well and with aching limbs it will not be that easy or fun.

That is to programming: start learning an easy programming language, do a little bit of learning every day, and eventually you will get it. Doing to much makes your brain tired and you may be so bored with the stuff that you may quit and do something else.

Increase the difficulty gradually

Having started the tracking experience, it may be boring to do the same path every time and after a while your fitness will tend to stagnate and adapt to what you are doing. That is, walk longer paths, and also more steep paths, the body tends to adapt to be able to handle the demands, but not more than necessary. We need to increase the work load to be able to handle more and more.When we are fitter, we can handle more difficult paths. However, even if it is good to challenge yourself a bit, do not overdo it (you may fall down and die). When you climb the mountain higher you will get a more panoramic view, it is beautiful and you can also start to see the path you took, and maybe also see different paths you can take to get to the same place.

That is to programming: Do not just learn the basics, they are good to have but can not help you in the long run, specially not if you want to have it as a profession. do the basics, repeat them, but when you feel comfortable with them, increase the amount of code you write, and also the difficulty of the code: that is, start doing design work, object oriented programming etc. Knowing the basics may help you write simple routines, but climbing higher will give you a better view of the surroundings and starting to put things together or how to do different designs to get to the same result.

Walk different mountains

No we may not have made it to the top of the first one yet, but does that really matter? Not really, the last part of getting to the top of a mountain may be easy, but it is usually not. Take Matterhorn, or Mount Everest for example. A lot of people climb them, but not all of them go for the top. you can go to the mountain and go as far as you have the time to do, or as high as you have the strength to do. this does not mean that you do not have a meaningful experience or doing a bad work. You doing as much as you can do or are willing to do, and even if its it not to the top, you may reach much higher than most people will do in their lifetime.

If you think that you did a mountain as good as you can or that you did it long enough to get bored. Try another one, it will give you another view, different tracks. It may have paths that is easier then the first mountain, but also paths that is harder. Maybe it will get you higher quicker and let you see more, maybe the paths are harder but the view is more rewarding than the first one.

That is to programming: Learn different programming languages, the first one you learn may not be a very good one to use professionally, but it may be a good language for learning basic programming. The new language may give you more opportunities to solve things quicker than the first one you learned. It may also be harder to do stuff in it, but when they are done, it may be better, quicker to do what you want it to do. for example Java may be easier to learn and use than C++ but performance wise, some C++ programs do better. On the other hand Java is more platform independent than C++, so it may be a better option for some companies. 
Different language can solve things easier than in other languages, or have better performance. the choice of language depends of the application you want to make. Learn more than one to widen your options and you will be better at your work.

So when do I get to the top?

Ok, lets be realistic. You may never get to the top. It depends on the mountain you decide to climb. Take Mount everest for example, it is the highest mountain we have, only a few made it to the top, and even fewer without injuries.

Take most mountains that is not a walk in the park, you may never reach the top, but you can get far enough to be satisfied with the experience and the rewards it give.

For programmers: not all programmers learn absolutely everything about one single programming language. they leave the top part for experts, that are fit to do the advanced stuff. 


Anyway programming is not really about doing the most advanced stuff or the most dangerous stuff. It is about doing stuff that other can follow. someone is coming after you and may have to maintain stuff you did. If it is to difficult they may not be able to go all the way, or understand how the heck you were able to get up to that cliff. programming is about leading the way for other to follow, you take a path that is fairly easy, sometimes not so easy but still manageable, you draw a map so that other can take the same path and continue from there.

So what are you waiting for, go out and climb a new mountain today!

Struts2 on Glassfish v3

OMG. It was not that trivial to get it going, at least not using Netbeans.

First trying to use maven since I need to have some EJBs, but I gave up after not being able to get the archetype for Struts2 do any project for me, first complaining about no commons-collections-3.2.1, downloading it and got a little bit longer, but then could not find anything for struts2.1.6.

So I went on creating a Struts2 project and got it to work by pointing out the jar files needed in Netbeans JavaEE webproject for Glassfish, YES there is a lot of jarfiles depending on each other since I also want to use TILES. Anyway, when I wanted the use the EJBs I had to create a EAR-project to hold everything together, what happens then?

Struts2 stops working, complaining of a lot of classes not loaded or cannot find. I spent almost all the afternoon before finally finding the problem.

Netbeans ant-script wants to copy every f...ing jar file that is not a TLD to the lib of the EAR, NOT into the Web applications WEB-INF/lib, which then makes sence for the errors in the log: when loading from that EAR/lib folder, it can not find struts-core classes since that jar file is in the Web-app because it includes TLDs,

So finally solved, hack the build-impl.xml and copy ALL jar files to the web-app so it works.Why Netbeans do this way I do not know, maybe someone can inform me?