Scala Stock Charts - part 6 some GUI things

I got a bit caught up in all the programming and finished up a few diagrams, then some refactoring and was able to remove a couple of hundred lines of code. The program now looks like this:

As you can see a lot have happened since the last pictures of the application. So here is some things.

Main part

The applications is build up around a main frame that contains the bar chart and some overlays. As you can see there is also some text fields and buttons. The upper part is to search stocks/indexes and also handle the overlays in the bar char. These fields and buttons are:

• Search field to enter a search criteria to filter the instruments to search for.
• Search button, to apply the search
• A combo box, to show the filtered search result, and also choose the stock to view. This combo box listen to changes and updates the bar chart with the selected stock
• Next there is a box to limit the range of the data. I have not introduced recalculation for splits and emissions so to have a possibility to handle to not show before these events, I added a box to filter away prices before this date part.
• Next there is the intervals for the moving averages in the chart, default they are set to 5,20,50 but can be changed but have as most 3 different, but also as few as one. Default they are Simple Moving Average (SMA),
• to change to Exponential Moving Average (EMA) there is a checkbox
• to update changed settings for Moving averages there is a button "Update averages"
• finally on the top there is a check box to switch on/off viewing of "Bollinger Bands"

In the bottom part there is buttons for additional chart types. these are the ones I picked as most interesting for me and they show some different aspect of the prices.

• Stocastics (show how a stocks price is doing in relation to past movements)
• RSI (shows how strong a stock is moving in the current direction)
• Aroon (shows if a stock is trending or oscillating)
• MACD (and also MACD-histogram) (a momentum oscillator)

As I mentioned the main part is a Bar chart diagram. It contains some overlays like volume, moving average and optional Bollinger bands. These help to get a basic overview of a stocks movement and trend. It is made up of a specialized panel for the bar chart to handle the drawing of the chart.

The main application implements a method named top that is used in wrapped Swing features used in Scala. I have an object representing my starting point of the application, the first lines looks like:
object ScaBorsApp extends SimpleSwingApplication with Logging{ def top = new MainFrame { title = "Technical Analysis for Stocks" val COL_GREEN = new Color(0,150,0) val searchField = new TextField { text = "a" columns = 10 } As you can see I provide a title and also here is the search field where to put a search string. The most important thing to note here is the inheritance of SimpleSwingApplication which makes this object a Swing application

The next part looks like this
/* * Search button, will get the pricelist and do filter and sort to present in combo box */ val searchBtn = new Button("Search") { action = Action("Search"){ val pricelists = PriceService.getPriceLists map { kv => kv.instruments } flatMap { case i: Instrument => i case x => x } filter { i: Instrument => i.name.toLowerCase().startsWith(searchField.text.toLowerCase) } val items = pricelists.toList sortWith {(i1,i2) => i1.name.toLowerCase < i2.name.toLowerCase} combo.peer.setModel( ComboBox.newConstantModel( for{p <- items} yield p.ticker + ": " + p.name ) ) } } /* * ComboBox to present filtered search for stocks, see searchButton action code how search is performed */ import event._ val combo = new ComboBox(List("-- Search to be able to select --")) { preferredSize = new Dimension(300,20) listenTo(selection) reactions += { case SelectionChanged(c) => curInstrument = new Instrument(selection.item.split(":")(0),selection.item.split(":")(1).trim,"") curPriceList = PriceService.getInstrumentData(curInstrument).filter(p => p.date.substring(0,fromTextBox.text.length).toInt >= fromTextBox.text.toInt) barchartPanel.updateList(curPriceList) case e => info(e.toString) } } ... val chkBollinger = new CheckBox { text = "Bollinger bands" reactions += { case x => barchartPanel.setBollinger(this.selected) } } Here you can see some on the fly declaration and reactions to events on the components.  The following code shows how to add these to the contents of other panels. you can also see the creation at the click of the button, creation of a generic diagram for "stochastics" mixed in with a DrawListener to make it drawable. Also how the series of stochastics is added to a generic diagram can be seen here.
/* * Top menu panel. */ val menu = new FlowPanel { contents += new Label("Stock") contents += searchField contents += searchBtn contents += combo contents += fromTextBox contents += avgSettings contents += chkExpAverage contents += btnUpdateAvg contents += chkBollinger } .... val bottomPanel = new FlowPanel{ contents += new Button(){ text = "Stochastics" reactions += { case bc: ButtonClicked => val stD = new Frame { title="Stochastics " + curInstrument.name } stD.open() val panel = new GenericDiagram with DrawListener val scrPan = new ScrollPane{ preferredSize = new Dimension(1180,220) contents = panel } stD.contents = scrPan val stoList = PriceUtil.stochastics(curPriceList, 14) panel.addSerie(ChartSerie(stoList,Color.blue)) panel.addSerie(ChartSerie(PriceUtil.slidingAvg(stoList, 3),Color.red)) panel.addDateSerie(curPriceList.map{p=>p.date}) panel.setLabelLevels(List(20d,50d,80d)) // need to force set max, it is not always 100 ?? panel.maxVal = 100d } } .... contents = new BorderPanel() { add(menu, BorderPanel.Position.North) add(scrollPane, BorderPanel.Position.Center) add(bottomPanel, BorderPanel.Position.South) } This last part of the code adds the actual panels containing the fields, bar chart and buttons to the Swing application.

Additional panels

To add some extra check to the main part there is some diagrams that first was their own specialized panel, but after some refactoring they now all except MACD uses a generalized panel called GenericDiagram and here is a part of that code:

And here is addition of some of the buttons into menu panel (top panel) followed by some example of the bottom panel with addition of button and reactions to its event. You see here that here is where the panel frame is set up inside the reaction to show the GenericDiagram with "stochastic" data.
val chkBollinger = new CheckBox { text = "Bollinger bands" reactions += { case x => barchartPanel.setBollinger(this.selected) } } ... trait BasicChartProps { val PAD_TOP,PAD_BOTTOM = 20 val PREF_H = 200 val PREF_W = 1180 var maxVal = Double.MinValue var minVal = Double.MaxValue var startX = 50 } /** * Generic panel for simple series */ class GenericDiagram extends Panel with BasicChartProps with Logging{ private var series = List[ChartSerie]() private var dateSerie = List[String]() ... Not so much to mention really except that as you see I extend with a trait with basic chart properties. this is intended to have common properties for this and other diagrams. The var's is to be able to add series and date series to the diagram. So this panel is only responsible to draw the things that are sent to it.

Ability to draw

No bar chart is complete if it is not possible to draw some help lines. This is managed by adding a trait for this, that can be mixed in wherever I want it. The code for that trais is:
... /** * Trait to use for Panels that should have line drawin possibilities */ trait DrawListener extends Panel { val pointZero = new Point(0,0) var curStart = pointZero//new Point(0,0) var curEnd = new Point(0,0) listenTo(mouse.clicks, mouse.moves, keys) reactions += { case e: MousePressed => curStart = e.point //moveTo(e.point) requestFocusInWindow() case e: MouseDragged => curEnd = e.point; repaint() //lineTo(e.point) case e: MouseReleased => moveTo(curStart);lineTo(e.point);curStart=pointZero;curEnd=pointZero case KeyTyped(_,'c',_,_) => path = new geom.GeneralPath repaint() case _ => repaint() } /* records the dragging */ var path = new geom.GeneralPath def lineTo(p: Point) { path.lineTo(p.x, p.y); repaint() } def moveTo(p: Point) { path.moveTo(p.x, p.y); repaint() } override def paintComponent(g: Graphics2D) = { super.paintComponent(g) g.setColor(Color.black) if(curStart.x != 0 && curEnd.y != 0){ g.drawLine(curStart.x,curStart.y,curEnd.x,curEnd.y) } g.draw(path) } } ... An example of this functionality is to draw help lines in a MACD diagram. (or any really)

This can be mixed in like in this case:
val panel = new MACDPanel(Nil) with DrawListener And then it is possible to draw on that panel.

Scala stocks part 5 - Refactoring, big time

There is soooo much stuff that I would like to look in to. Alfresco, Liferay, GWT, Spring, Scrum, Agile etc. Right now though, I continue with my Scala project doing a Technical analysis tool for stocks using a Swing gui, also that with Scalas Swing libraries.

After working with the GUI, I started to get annoyed on myself mixing things up here and there, so I decided to refactor a bit. Actually, since I wrote the last post, I got a bit further then just plain refactoring. Simultaneously with the refactoring I worked on the GUI and also restructured code to finish up the loading of prices, enhancement of that loading and in that work I also managed to introduce some "Actors".

I will now also start to explain a little bit more what the code does.

I will split the refactoring and explanation in more than this post, otherwise I may get too long.

Refactoring of the FileHandler.
The new file is now much shorter, and includes only stuff to handle files. The extraction and transformation to the model is removed from the file handling. that messy code can be seen here http://ironicprogrammer.blogspot.com/2011/08/scala-stock-charts-part-3-saving-quotes.html if you have not already read it.

FileHandler.scala

/* @author Peter Johansson / Ironic programmer */ package me.ironic.scabors.file import me.ironic.scabors.model._ import me.ironic.scabors.util.Logging object FileHandler{ def apply() = { new FileHandler() } } class FileHandler extends Logging { import java.io.{File,BufferedWriter,FileWriter,IOException} import scala.collection.mutable.{ HashMap => MutableHashMap} import scala.collection.immutable.{ HashMap } import scala.io.Source val rootDir = { scala.util.Properties.userDir } val filesDir = { rootDir + "/files"} val logFile = { rootDir + "/" + "sBors.log"} val pricelistsFile = { rootDir + "/files/pricelists.txt"} def checkDir(dirName: String){ val f = new File(dirName) if(!f.exists){ f.mkdirs info("!! Created dir at : " + dirName) } } def writeToFile(fileName: String, append: Boolean, rows: List[String]){ // Check that dir exista or creat if not checkDir(filesDir) val bw = new BufferedWriter(new FileWriter(fileName,append)) rows foreach { row => bw.write(row + "\n") } bw.close() } def priceFile(ticker: String) = { rootDir + "/files/" + ticker.replace(" ","_") + ".txt" } def loadFile(fileName: String) = { val fc = Source.fromFile(fileName) fc.getLines.toList } }
In the start of the file, you see:
object FileHandler{ def apply() = { new FileHandler() } } This is a companion object I use it to have a factory method. Maybe not really nescessary but at least I get away with not having to write
new FileHandler()
every time I need it and can use the shorter version
FileHandler()

The rest of the FileHandler should be quite easy to figure out. I have only one write-method where I send in the filename, if I want to append and a list of strings to write to the file. With this I can do all the saving I need.

There is also only one method for getting the file content (loadFile), this returns a list of strings which I later can traverse and to all the logic with.

I also have a method to have a format for how a price file is named. I do not want any blanks in the filename so this method is replacing that with "_" instead.

As you can see the FileHandler extends Logging. I do not need that here since no logging is going on in this class as it is now, but I can add it if I want. More about the Logging later.

Introducing PriceDAO.
To hide FileHandler from the rest of the system, and also to the mapping between files and model that was before done in the FileHandler, I introduced a DAO object to do the work. Doing this I can always change the way I persist later without too much impact on the code.

There is still some refactoring to do in this one since I have not changed the fact that I use mutable lists.

PriceDAO.scala

/* @author Peter Johansson / Ironic programmer */ package me.ironic.scabors.dao /** * Provides service methods and communicate with underlaying storage */ import me.ironic.scabors.util.Logging import me.ironic.scabors.model._ import me.ironic.scabors.file.FileHandler import scala.collection.mutable.{ HashMap => MutableHashMap} import scala.collection.immutable.{ HashMap } object PriceDAO extends Logging{ def saveDailyPrices(prices: List[PriceItem]) = { info("## Saving daily prices") var lists = new MutableHashMap[String,List[Price]] var curList = "?" prices foreach { price => price match { case q: Price => { savePriceToFile(q); lists(curList) = List(q) ::: lists(curList) } case qList: PriceList => { curList = qList.name; lists += qList.name -> List() } case _ => warn(price.toString) } } // Save current lists to file... saveStockPriceList(lists) } /** * Method for saving a price to file */ protected def savePriceToFile(price: Price) = { val fileHandler = FileHandler() val fName = fileHandler.priceFile(price.ticker) fileHandler.writeToFile(fName, true, List(price.toString)) } /** * Method for saving an updated pricelist map to file. */ protected def saveStockPriceList(pricelist: MutableHashMap[String,List[Price]]){ info("## Saving pricelists") val sb = new StringBuffer sb.append("{ \"pricelists\": [\n") for{ (listname,prices) <- pricelist } { sb.append("\t{ \n\t\t\"listname\": \"" + listname + "\",\n") sb.append("\t\t\"stocks\": [\n") for{ price <- prices } { sb.append("\t\t\t{ \"ticker\": \""+price.ticker + "\", \"aktie\": \"" + price.aktie + "\"},\n") } sb.append("\t\t],\n") sb.append("\t},\n") } sb.append("}\n") val fileHandler = FileHandler() val fName = fileHandler.pricelistsFile fileHandler.writeToFile(fName, false, List(sb.toString)) } /** * Method for loading price lists to a HashMap, * each entry in the map is a list of Instruments */ def loadPriceLists : MutableHashMap[String, List[Instrument]] = { info("## Loading pricelists") var pricelists = new MutableHashMap[String, List[Instrument]] val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r /* * When hitting a PriceList, a new list begins */ var curList = "?" val fileHandler = FileHandler() val fName = fileHandler.pricelistsFile val rows = fileHandler.loadFile(fName) rows foreach { line => line match { case ListExtractorRE(listname) => val pList = new PriceList(listname) curList = pList.name pricelists += pList.name -> List() case InstExtractorRE(ticker, aktie) => val inst = new Instrument(ticker, aktie) pricelists(curList) = List(inst) ::: pricelists(curList) case _ => None } } pricelists } /** * Method for loading all prices for a specific instrument. */ def loadInstrumentData(inst: Instrument) : List[Price] = { val fileHandler = FileHandler() info("Loading instrument " + inst) val fName = fileHandler.priceFile(inst.ticker) val PriceExtractorRE = """Price\(([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*).*\)""".r val pricelines = fileHandler.loadFile(fName) val prices = pricelines map { line => line match { case PriceExtractorRE(date,ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) => Price(date,ticker,aktie,diffKr.toDouble,diffPerc.toDouble,buy.toDouble,sell.toDouble,last.toDouble,high.toDouble,low.toDouble,returnNo.toDouble,returnKr.toDouble) } } prices.toList } }
Some explanation maybe?

Well, I am not going into every line. I choose some parts that I think is interesting parts of Scala.
def saveDailyPrices(prices: List[PriceItem]) = { /*....*/ var lists = new MutableHashMap[String,List[Price]] var curList = "?" prices foreach { price => price match { case q: Price => { savePriceToFile(q); lists(curList) = List(q) ::: lists(curList) } case qList: PriceList => { curList = qList.name; lists += qList.name -> List() } case _ => warn(price.toString) } }
This code it possibly a good candidate to do some further refactoring on to try to make it immutable, but this is what is going on.

• lists is mutable, because I need to add things in it, and I assume that I do not know what lists are coming in. (Even if they are pretty known from the file)
• prices is a list of type List[PriceItem], this means that everything in that list is of something under type PriceItem but right now I am only interested in the types Price and PriceList
• If it is a PriceList I know that a new price list is starting, I add this as a map key with an empty list as the map value.
• The flow goes on and for every Price that comes up this is added in the current list.

The next chunk of interest...
val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r /*....*/ val rows = fileHandler.loadFile(fName) rows foreach { line => line match { case ListExtractorRE(listname) => val pList = new PriceList(listname) curList = pList.name pricelists += pList.name -> List() case InstExtractorRE(ticker, aktie) => val inst = new Instrument(ticker, aktie) pricelists(curList) = List(inst) ::: pricelists(curList) case _ => None } }
Here is the loading of the price lists. I will sow you part of the file so you know what we are dealing with.
{ "pricelists": [ { "listname": "OMX Stockholm Mid Cap", "stocks": [ { "ticker": "ORES", "aktie": "<d6>resund"}, { "ticker": "AF B", "aktie": "<c5>F B"}, { "ticker": "VNIL SDB", "aktie": "Vostok Nafta Investment"}, { "ticker": "WIHL", "aktie": "Wihlborgs"}, { "ticker": "WALL B", "aktie": "Wallenstam B"},

That is how it is saved. So seeinging that makes it easier to explain what the above code does.

• first there is 2 "extractors", these are regular expressions. In Scala you can use the 3-qouted strings to make a regular expression just by putting .r after it. This expressions can then be used as types cases in pattern matching and is very powerful way to extract thing.
• (""".*"listname": "(.+)",.*\s*""".r) is for extracting the name of a list. This matches on the line in the file beginning with any character and then "listname":
• the next "extractor" is for matching an instrument. for convenience I write the file with a "ticker" on a new line to avoid complex regexps. So I only need to parse the file and match on these.
• When matching a new list I put a new map entry with an empty list, and when hitting a "ticker" I just add that instrument in the current list. Similar to what is going on when saving them.

Major restruction of PriceLoad.

I more or less totally reworked the loading of the prices from the site they are available from

One thing I wanted to do was to use actors. Here is were I got loose on doing that. Well, thats a bit exaggerating since there is only one Actor involved here (and another one in the Logging) so not so big fuzz. Here is the code, and I get into some details after it.

PriceLoad.scala

/* @author Peter Johansson / Ironic programmer */ package me.ironic.scabors.dao import me.ironic.scabors.model._ import me.ironic.scabors.util._ // Singleton to handle loading of Prices object PriceFetcher extends Logging { import java.util.Calendar import java.text.SimpleDateFormat val fmt = new SimpleDateFormat("yyyyMMdd") def stringDateList(c: Calendar,bDate:String) : List[String] ={ if(fmt.format(c.getTime()) == bDate) { List(fmt.format(c.getTime())) } else { val c2 = c.clone().asInstanceOf[Calendar] c2.add(Calendar.DATE,1) fmt.format(c.getTime()) :: stringDateList(c2,bDate) } } def main(args: Array[String]) = { if(args.length < 1 || args.length > 2){ println("Usage: PriceFetcher YYYYMMDD # for single day") println("Usage: PriceFetcher YYYYMMDD YYYYMMDD # for batch mode") sys.exit(1) } val list = { if(args.length == 1){ List(args(0)) } else { require(args(1).compareTo(args(0)) > 0) val dFrom = fmt.parse(args(0)) val cal = Calendar.getInstance() cal.setTime(dFrom) stringDateList(cal, args(1)) } } info("Will run fetching for list " + list) val fetcher = new PriceFetchActor fetcher.start list foreach { s => fetcher ! Daily(s) } fetcher ! Stop // Need to stop the logger, otherwise there is a hanging thread import scala.actors.Actor.State while(fetcher.getState != State.Terminated){ // to wait until all work is done } stopLogger sys.exit(0) } } import scala.actors.Actor import scala.actors.Actor._ import me.ironic.scabors.util.Logging class PriceFetchActor extends Actor with Logging{ import scala.io.Source val baseUrl = "http://thesite/" def act(){ loop{ react { case d: Daily => info("Fetching prices for " + d) val uri = baseUrl + d.date + "k.txt" try{ val lines = readUrl(uri) val handledLines = processLines(lines,d.date) info("Did lines : " + handledLines.size) PriceDAO.saveDailyPrices(handledLines) } catch { case x => warn(x.toString) } case Stop => info("Stopping PriceFetchActor") exit() case _ => } } } def readUrl(url: String) : List[String] = { val fc = Source.fromURL(url).getLines().toList fc } val PriceExtractorRE = """([[^Ticker]^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t.*""".r val ListExtractorRE = """(OMX[^\t]+)\t+""".r // Should go through the lines from the file and make a list with jSon-like format. def processLines(lines: List[String],date: String) : List[PriceItem] = { info("-> Processing lines") val instruments = lines takeWhile(s=> ! s.startsWith("Externa listan")) instruments map { inst => inst match { case ListExtractorRE(list) => PriceList(list)//"{\"instrumentList\": \"" + list + "\"}" case PriceExtractorRE(ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) => Price(date,ticker,aktie,diffKr.replace(",",".").toDouble,diffPerc.replace(",",".").toDouble,buy.replace(",",".").toDouble,sell.replace(",",".").toDouble,last.replace(",",".").toDouble,high.replace(",",".").toDouble,low.replace(",",".").toDouble,returnNo.replace(",",".").toDouble,returnKr.replace(",",".").toDouble) case _ => InvalidPrice(date,inst.replace("\t","[tab]")) } } } }
This is an application object with a main method so it is available to run from the command line. What it does is to check if you enter with one or two arguments. Entering with one argument will load for one day. Entering with two arguments will run for a date span. I do not really car if there is days that does not have any prices, the only thing that will happen is that a FileNotFoundException will be logged.

The date span wil be recursively calculated to a list of strings with the method stringDateList.

After getting the list with dates, whether it is one item or more, I am traversing that list and sending a Daily object to an actor to do the loading and saving of that day. When all dates are handled I send a Stop message so the actor can exit itself.

Since I am sending to actors the flow of the program just passes to the end of the main method so I have to check the state of the actor before I do the exit. (There is a hang otherwise, because there is a thread that is still running somewhere.) and I do not want to do a sys.exit() before since that kills the thread that is loading.

The messages that is sent is case object/class that is defined like:
case object Stop case class Daily(date: String)
Other than the actor logic, the load and save is more or less the same as before. Just that now it is threaded and runs by an actor by sending messages to it.

Well, that was enough for this time. The GUI is more evolved also, but that is another story, another day. But a teaser to show how it looks so far:

Scala stock charts, part 4 - Woking with the GUI - 1

I have been busy the last couple of weeks, or better saying, too lazy to sit home with my Scala Technical Analysis tool. Anyway, this weekend I decided to continue and this time to start from the GUI and start using the functionality done so far from a GUI.

I have also done som restructuring in the packages which now is:

package me.ironic.scabors.app
package me.ironic.scabors.service
package me.ironic.scabors.model
package me.ironic.scabors.file

even if I have not reworked the messy file-package. I will do that later. however, I finished up with loading the price list, and that code now look like this:

def loadPriceLists : MutableHashMap[String, List[Instrument]] = { log("## Loading pricelists") var pricelists = new MutableHashMap[String, List[Instrument]] val fc = Source.fromFile(pricelistsFile) val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r /* * When hitting a PriceList, a new list begins */ var curList = "?" fc.getLines foreach { line => line match { case ListExtractorRE(listname) => val pList = new PriceList(listname) curList = pList.name pricelists += pList.name -> List() case InstExtractorRE(ticker, aktie) => val inst = new Instrument(ticker, aktie) pricelists(curList) = List(inst) ::: pricelists(curList) case _ => None } } return pricelists }
I also broke out the model classes from the file package, and put it in another file called PriceModel.scala, you can also see that I am now using "Price" which is a better name than "Quote" so all references to Quote is now replaced with Price. The content of that file is:

PriceModel.scala

package me.ironic.scabors.model // To be able to handle Iterator[PriceItem] trait PriceItem case class InvalidPrice(date: String, content:String) extends PriceItem case class PriceList(name:String) extends PriceItem case class Price(date: String, ticker :String, aktie : String, diffKr : Double, diffPerc : Double, buy : Double, sell : Double, last : Double, high : Double, low : Double, returnNo : Double, returnKr : Double ) extends PriceItem case class Instrument(ticker:String, aktie: String)

I also made a Service object so that the GUI does not need to know or depend on anything from the file package. So far as I have gotten with the GUI it just contains two methods, one to get all price lists and one to get the data for a chosen stock. Here is the code:

PriceService.scala

package me.ironic.scabors.service object PriceService { import me.ironic.scabors.file.FileHandler val pricelists = { println("# loading pricelists") FileHandler.loadPriceLists } def getPriceLists = { // using already loaded pricelists, i.e loading is done in val... pricelists } def getInstrument(inst: String) = { FileHandler.loadInstrument(inst) } }

GUI programming done in Scala

I decided to leave the idea of using Java FX that I had from the beginning and that I wrote in the beginning post. Instead I decided to use Scala Swing. Not so easy as it turned out, because first of all, it has been a couple of years since I last did any Java Swing programming and second of all, it was not very similar and not easy to transform Java Swing thinking to Scala Swing thinking. Well, it is less code, but surly, at this stage it looks really messy. Maybe it takes a while for my brain to melt Scala Swing programming and the fact that google got warm from all searching and few good examples does not help.

I am to tired to explain most of the code, maybe in a later post, but here is what the purpose if the GUI is so far.

• Provide a text field to filter search for stocks "Instrument"
• Provide a button to perform the search
• Provide a ComboBox to present the filtered result from the search and to select stock to draw
• Provide Panel for drawing a Bar Chart of selected stock

I have implemented and it works as I wand, except that the tedious job to provide the functionality to draw a bar chart is not done. Thea is a later exercise and a future post to describe. I let you see the messy code for this. 

ScaBorsApp.scala

package me.ironic.scabors.app import swing._ import java.awt.Dimension import me.ironic.scabors.service._ import me.ironic.scabors.model._ object ScaBorsApp extends SimpleSwingApplication{ def top = new MainFrame { title = "Technical Analysis for Stocks" val searchField = new TextField { text = "a" columns = 20 } /* * Search button, will get the pricelist and do filter and sort to present in combo box */ val searchBtn = new Button("Search") { action = Action("Search"){ val pricelists = PriceService.getPriceLists map { kv => kv._2 } flatMap { case i: Instrument => i case x => x } filter { i: Instrument => i.aktie.toLowerCase().startsWith(searchField.text.toLowerCase) } val items = pricelists.toList sortWith {(i1,i2) => i1.aktie.toLowerCase < i2.aktie.toLowerCase} combo.peer.setModel( ComboBox.newConstantModel( for{p <- items} yield p.ticker + ": " + p.aktie ) ) } } /* * ComboBox to present filtered search for stocks, see searchButton action code how search is performed */ import event._ val combo = new ComboBox(List("-- Search to be able to select --")) { preferredSize = new Dimension(300,20) listenTo(selection) reactions += { case SelectionChanged(c) => curInstrument = new Instrument(selection.item.split(":")(0),selection.item.split(":")(1).trim) curPriceList = PriceService.getInstrument(curInstrument.ticker) barchartPanel.updateList(curPriceList) } } var curInstrument: Instrument = Instrument("---","Not choosen") var curPriceList: List[PriceItem] = Nil /* * Top menu panel. */ val menu = new FlowPanel { contents += new Label("Stock") contents += searchField contents += searchBtn contents += combo } val barchartPanel = new BarChartPanel(curPriceList) val scrollPane = new ScrollPane { preferredSize = new Dimension(1180,700) contents = barchartPanel } contents = new BorderPanel() { add(menu, BorderPanel.Position.North) add(scrollPane, BorderPanel.Position.Center) } preferredSize = new Dimension(1200,800) } } class BarChartPanel(var list: List[PriceItem]) extends Panel { import java.awt.Color background = Color.white preferredSize = new Dimension(1180,680) override protected def paintComponent(g: Graphics2D) { super.paintComponent(g) } def updateList(newList: List[PriceItem]) = { list = newList println("Updating list : " + list) } }
Yeah, thats right, I also gave up SBT, I am now using gradle to build. here is the code for that. build.gradle

build.gradle

apply plugin: 'scala' repositories { mavenCentral() } dependencies { // Libraries needed to run the scala tools scalaTools 'org.scala-lang:scala-compiler:2.9.1' scalaTools 'org.scala-lang:scala-library:2.9.1' // Libraries needed for scala api compile 'org.scala-lang:scala-library:2.9.1' // Libraries needed for scala.swing // scalaTools 'org.scala-lang:scala-swing:2.9.1' compile 'org.scala-lang:scala-swing:2.9.1' } task createSourceDirs << { println "# Creating source dirs" ['src/main/scala','src/test/scala','src/main/java','src/test/java'].each { println " - $it" new File(it).mkdirs() } }
Until later..... ciao!

Web Fragment using Scala

Do you want to use Scala in a Java web environment, or write web fragment using scala. Or maybe you would ask: why do it at all? The answer is, because you can!

Well my little spike succeeded without any severe headache, actually so well that I did not got any at all and there was never any risk to get one.

I used SBT to build my Scala code, probably I could use it to make the full application, but I used Netbeans to create a web projet for my spike.

Setting up for SBT.

create folder for the project, for example:
>mkdir webfrag
enter the directory and create and edit build.sbt for servlet programming
>touch build.sbt

Put the following in the build-sbt file
name := "WebFrag" version := "1.0" organization := "me.ironic" scalaVersion := "2.9.1" scalaSource in Compile <<= baseDirectory(_ / "src") scalaSource in Test <<= baseDirectory(_ / "test") libraryDependencies += "javax" % "javaee-api" % "6.0" % "provided" libraryDependencies += "log4j" % "log4j" % "1.2.16" % "provided" libraryDependencies += "junit" % "junit" % "4.8.2" % "test"
Create a Servlet in Scala in this way.
create src/ folder and create and edit a file for the servlet, for example src/WebFragHello.scala

You can write what you want, but to make a form of HelloWorld-like example fill it with for example:

package me.ironic import javax.servlet.annotation._ import java.io._ import javax.servlet._ import http._ @WebServlet(name = "WebFragHelloServlet", urlPatterns = {Array("/hello")}) class WebFragHelloServlet extends HttpServlet { override def doGet(req: HttpServletRequest, res: HttpServletResponse) : Unit = { val user = req.getParameter("user") match { case s: String => s case _ => "John Doe" } val out = res.getWriter() out.println(toHtml(user).toString()) } def toHtml(user: String) = <html> <body> <h1>Hello {user}</h1> </body> </html> }
This code also "catches" the case that user parameter is not sent.A special feature seams to be that the scala code does not understand urlPattern as used in Java in the annotation, that is why I use {Array("/hello")} instead of {"/hello"}

Compile and package the code with SBT
>sbt compile
>sbt package

To make it work as a web-fragment: Put the compiled jar in WEB-INF/lib folder in a Servlet 3.0 server, you also need to put scala-library.jar in the server to be able to use scala objects. In netbeans you just need to add that jar-file to "libraries" folder in the test web project

Surf to .../context-root/hello or /context-root/hello?user=Ironic

Neet isn't it?

Scala stock charts, part 3 - Saving the quotes

Going further in programming the functionality behind the technical analysis tool for stocks made in Scala we are going to save the daily quotes to separate file, we are also going to supply functionality to load the data for a stock, and save and read a price list divided in 3 lists.

The format of the files is a bitt different, the Quotes I will save and read as they are, but the price list will have a different format that will look as jSon and is saved in a way that will be easy to read with pattern matching. For the matter of fact also the Stock-quotes for a single instrument will be read and transformed to a list of Quote's with pattern matching.

FileHandler.scala

/* @author Peter Johansson / Ironic programmer */ package scala.ironic.file object FileHandler { import java.io.{File,BufferedWriter,FileWriter,IOException} import scala.collection.mutable.{ HashMap => MutableHashMap} import scala.collection.immutable.{ HashMap } import scala.io.Source val rootDir = { scala.util.Properties.userDir } val filesDir = { rootDir + "/files"} val logFile = { rootDir + "/" + "sBors.log"} val pricelistsFile = { rootDir + "/files/pricelists.txt"} def saveDailyQuotes(quotes: Iterator[QuoteItem]) = { log("## Saving daily qoutes") log("--- rootDir : " + rootDir) log("--- filesDir : " + filesDir) // Check that dir exista or creat if not checkDir(filesDir) var lists = new MutableHashMap[String,List[Quote]] var curList = "?" quotes foreach { quote => quote match { case q: Quote => { saveQuoteToFile(q); lists(curList) = List(q) ::: lists(curList) } case qList: QuoteList => { curList = qList.name; lists += qList.name -> List() } case _ => log("! " + quote) } } // Save current lists to file... saveStockPriceList(lists) } def checkDir(dirName: String){ val f = new File(dirName) if(!f.exists){ f.mkdirs log("!! Created dir at : " + dirName) } } def saveQuoteToFile(quote: Quote) = { val fileName = scala.util.Properties.userDir + "/files/" + quote.ticker.replace(" ","_") + ".txt" val bw = new BufferedWriter(new FileWriter(fileName,true)) bw.write(quote + "\n") bw.close() } def saveStockPriceList(pricelist: MutableHashMap[String,List[Quote]]){ log("## Saving pricelists") val bw = new BufferedWriter(new FileWriter(pricelistsFile,false)) val sb = new StringBuffer sb.append("{ \"pricelists\": [\n") for{ (listname,quotes) <- pricelist } { sb.append("\t{ \n\t\t\"listname\": \"" + listname + "\",\n") sb.append("\t\t\"stocks\": [\n") for{ quote <- quotes } { sb.append("\t\t\t{ \"ticker\": \""+quote.ticker + "\", \"aktie\": \"" + quote.aktie + "\"},\n") } sb.append("\t\t],\n") sb.append("\t},\n") } sb.append("}\n") bw.write(sb.toString) bw.close() } def log(msg: String){ println(msg) val bw = new BufferedWriter(new FileWriter(logFile,true)) bw.write(msg + "\n") bw.close() } def loadPriceLists : HashMap[String, List[Instrument]] = { log("## Loading pricelists") val pricelists = new HashMap[String, List[Instrument]] val fc = Source.fromFile(pricelistsFile) val ListExtractorRE = """.*"listname": "(.+)",.*\s*""".r val InstExtractorRE = """.*"ticker": "(.+)", "aktie": "(.+)".*\s*""".r fc.getLines foreach { line => line match { case ListExtractorRE(listname) => println("#### got listname : " + new QuoteList(listname)) case InstExtractorRE(ticker, aktie) => println("# got instrument : " + new Instrument(ticker, aktie)) case _ => } } return pricelists } def loadInstrument(ticker: String): List[QuoteItem] = { val fn = rootDir + "/files/" + ticker.replace(" ","_") + ".txt" val fc = Source.fromFile(fn) val QuoteExtractorRE = """Quote\(([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*),([^\,]*).*\)""".r println(QuoteExtractorRE) val quotelines = fc.getLines// filter( s=> ! s.startsWith("Quote")) val quotes = quotelines map { line => line match { case QuoteExtractorRE(date,ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) => Quote(date,ticker,aktie,diffKr.toDouble,diffPerc.toDouble,buy.toDouble,sell.toDouble,last.toDouble,high.toDouble,low.toDouble,returnNo.toDouble,returnKr.toDouble) case _ => InvalidQuote("?", line) } } return quotes.toList } } That is a lot of code that I wrote on the fly, but not as much as it would have been to do the same thing in Java. the nice thing is the pattern matching, that saves a lot pain that Java code would bring, for example reading every line in the file would not be more code, but also creating and getting the values to objects would give a lot of more code.

[coming... Java code to do it as a comparison]

checkDir: is a method to create directory structure if it does not exist. Actually the first time loading the quotes there is no /files dir if it is not created manually, to avoid crashing, and open upp for the possibility to later use some config to have the data files in a directory of choice this is to ensure that the supplied configured dir will be created.

saveDailyQuotes: traverse the content of a daily quote file and calls to save single quote and collect the price lists and calls to save these.

saveStockPriceList: saves the current instruments divided in their lists to a file.

saveQuoteToFile: append a single quote to its data file.

Finally the methods for loading price lists and a single instrument

loadPriceLists: loads the pice lists back to a map with a list name and a list of Instruments. (Actually it is note yet finished, since I want to think out a way to avoid having a mutable HashMap, now it only prints out what it gets.)

loadInstrument: loads the file of a single stock to a list of Quote's


I am not completely happy with it, there is some duplication of code, for example I create new Writers in every method that save things. There is certainly things that can improve here. However, refactoring is a later issue, and this does what I want it to do, so for now it will have to do.

Now the interesting part is beginning to appear, next thing is to make functionality to take car of the technical stuff for the charts to be drawn.

Scala stock charts, part 2 - Getting the quotes

I am Swedish so I am for the moment only interested in the Swedish stock market. I have found a site where to get the quotes, it is in text format and I intend to read this information and save it in in other files, one file for each instrument (I call on single stock instrument since it is a form of financial instrument).

The format of the daily file is: (There is an example file to work with at : http://sites.google.com/site/ironicprogrammer/files/20110817k.txt)

Slutkurser från OMX 2011-08-17 Slutkurser från Burgundy 2011-08-17
Ticker Aktie +/- +/-% Köp Sälj Senast Högst Lägst Oms (antal) Oms (SEK) Senast Högst Lägst Oms (antal) Oms (SEK)
OMX Stockholm Large Cap
ABB ABB Ltd -2,30 -1,66 136,40 136,50 136,40 139,40 136,40 3299500 453921900 136,6 139,4 136,6 183000 25240000

It is hard to see in that little box, however. first row is title information of the file.
Slutkurser från OMX 2011-08-17[6 tabs]Slutkurser från Burgundy 2011-08-17

That is not really necessary and can be skipped as the next line, but the next line is interesting to see the structure of the quotes:

Ticker Aktie +/- +/-% Köp Sälj Senast Högst Lägst Oms (antal) Oms (SEK)[6 tabs]Senast Högst Lägst Oms (antal) Oms (SEK)

 When reading the file I will only be interested in the quotes from OMX, the Burgundy I do not want so when saving the quotes to different files I will not consider these, so it is the blue marked fileds that would be used.

The third row

OMX Stockholm Large Cap

is the name of the first list of quotes. It continues like this in the file;
Listname
Ticker Aktie ...
...
Listname ...

Ticker Aktie ...
...

I am only interested in the first three lists, the forth is named "Externa listan", and I will use this name to break the read.

So lets start write som code (finally!!)

I use the REPL for a little bit trial before I put in in classes/objects.

To test to read the file:

scala> val url = "http://sites.google.com/site/ironicprogrammer/files/" url: java.lang.String = http://sites.google.com/site/ironicprogrammer/files/ scala> val file = "20110817k.txt" file: java.lang.String = 20110817k.txt scala> import scala.io.Source import scala.io.Source scala> val fc = Source.fromURL(url+file) fc: scala.io.BufferedSource = non-empty iterator scala>; fc.getLines foreach { println } Slutkurser fr?n OMX 2011-08-17 Slutkurser fr?n Burgundy 2011-08-17 Ticker Aktie +/- +/-% K?p S?lj Senast H?gst L?gst Oms (antal) Oms (SEK) Senast H?gst L?gsOms (antal) Oms (SEK)<br /> OMX Stockholm Large Cap ABB ABB Ltd -2,30 -1,66 136,40 136,50 136,40 139,40 136,40 3299500 453921900 136,6 139,4 136,6 183000 25240000<br /> ALFA Alfa Laval -0,80 -0,68 117,50 117,60 117,50 119,10 114,80 1121500 131304100 117,6 118,9 114,49223 5760000

Ok, everything seams to work so this need to be taken car of in some other way.

Lets put it in a bit more organized way:

I put a file in src/main/scala/ironic that is named QuoteLoad.scala with the following content:

QuoteLoad.scala

/*
 @author Peter Johansson / Ironic programmer 
 */

package scala.ironic

// To be able to handle Iterator[QuoteItem]
trait QuoteItem 

case class InvalidQuote(date: String, content:String) extends QuoteItem
case class QuoteList(name:String) extends QuoteItem
case class Quote(date: String, ticker :String, aktie : String,
    diffKr : Double, diffPerc : Double, buy : Double,
    sell : Double, last : Double, high : Double,
    low : Double, returnNo : Double, returnKr : Double
  ) extends QuoteItem

// Singleton to handle loading of quotes
object QuoteLoad {
   def main(args: Array[String]) = {
      
      loadQuotes("http://sites.google.com/site/ironicprogrammer/files/","20110817")
   }
   
   val QuoteExtractorRE = """([[^Ticker]^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t.*""".r
   val ListExtractorRE = """(OMX[^\t]+)\t+""".r
  
   def loadQuotes(url: String, date: String){
      import scala.io.Source
      
      val uri = url + date + "k.txt"
      val fc = Source.fromURL(uri) 

      val instruments = processLines(fc.getLines(),date)
      println (instruments)
      instruments foreach { inst => inst match {
        case ilist: QuoteList => 
            println(ilist)
        case q: Quote =>
            println(q)
        case _ => println(inst)//
        }
      }
   }
  
    // Should go through the lines from the file and make a iterable with QuoteItem's. 
    def processLines(lines: Iterator[String],date: String) : Iterator[QuoteItem] = {
      val instruments = lines takeWhile(s=> ! s.startsWith("Externa listan")) 
      instruments map { inst =>  
         inst match {
          case ListExtractorRE(list) => QuoteList(list)//"{\"instrumentList\": \"" + list + "\"}"
   case QuoteExtractorRE(ticker,aktie,diffKr,diffPerc,buy,sell,last,high,low,returnNo,returnKr) 
            => Quote(date,ticker,aktie,diffKr.replace(",",".").toDouble,diffPerc.replace(",",".").toDouble,buy.replace(",",".").toDouble,sell.replace(",",".").toDouble,last.replace(",",".").toDouble,high.replace(",",".").toDouble,low.replace(",",".").toDouble,returnNo.replace(",",".").toDouble,returnKr.replace(",",".").toDouble) 
          case _ => InvalidQuote(date,inst.replace("\t","[tab]"))
        } 
      } 
    }
    
}

Ok there may be some features to explain here. 
First of all, I make use of an singleton object here so I will be able to call from a terminal window. So it has a main method, but I also extract the functionality so the future client will be able to use the same methods to load quotes.

There is a trait so I can avoid getting Iterator[Any] and instead getting Iterator[QuoteItem] to be able to handle the case class objects only, when I later will save the quotes to different files.

QuoteList: is for holding the name of a list (there is different list of quotes according to its category, that is more or less the amount of trading that is done and the size of the companies)
Quote: is for holding the quote for the day for a specific share
InvalidQuote: is for holding (and logging for error checking) an invalid row from the file.

There is two regexp's that is handled later in pattern matching to extract the lists and quotes from the file. They look for the structure in the file for valid quotes and the simpler one is for extracting the list name that is only one text.

def loadQuotes(url: String, date: String): handles the load from the URL and calls a helper method to extract the valid quotes and lists.

This will have to do for a first version, later I will save these quotes to different files.

Scala stock charts, part 1 - Thoughts about structure

Having read through almost halfway, and not even coming to the functional part, in the online book "Programming Scala" by Alex Payne (http://ofps.oreilly.com/titles/9780596155957), I am starting to get some ideas of what I want to try out in my new Technical Analysis program for stocks.  For sure I would like to use SBT (https://github.com/harrah/xsbt/wiki) as a building tool and try to use "Specs" for tests (http://ofps.oreilly.com/titles/9780596155957/ScalaToolsLibs.html#ScalaSpecs).

To start with I want to build my program using either common classes or case classes, haven´t clearly figured that out, at least I want to use companion objects. Anyway, I want to get in touch with different features in Scala so there probably will be some file handling with regexp in Scalas pattern matching. At the same way as I want to get in touch with as many features as possible, I also want to keep it as simple as possible so there probably will be som rewrites during the course of my little project as I learn more. If possible I will use trait also, but probably after a rework. I will start more "java-like" with more OO-structure and then try to "functionalize" it.

I haven´t really figured out if I want o use JavaFX either. So I will try to build a library in Scala that ccan be used by client of choice, may be JavaFX but maybe also Flex or Swing. That I will figure out later.

More about this later....
and then probably with some code :)

Long time no see

It has been a while since last post. Sorry about that! It has been a period of organizational problems (not as fun as programming problems that needs to be solved) that could have been written about, but to sit in the middle of a stormy change takes the energy away.

Anyway, this is what I have been doing:
I started looking at Scala. For me it looks really interesting and I think it is something that is worth looking in to since it joins well with Java and runs on the JVM. You also need to write less code to do the same thing as with Java.
To learn Scala I have small idea that I should take my old Java-Swing tool for technical analysis of stocks on the Swedish stock market, set it up to use Scala with a JavaFX-client for the GUI. In this way I intend to get into Scala and at the same time JavaFX a little bit. this little tool of mine I have had as a project to learn new programming languages for over 10 years now. From the first version in an Excel spreadsheet it has taken the way over Turbo-pascal, Visual C++, maybe I also had a C#-client for a while until I returned to the Java Swing client.

I also started to read a lot of agile books about modelling and developing and also a bit about Neuro Linguistic Programming but that is a completely different subject but may be good if you want to take a leading role and/or how to handle the users and other team members and just get better at asking questions as one area of self development.

I will go on, so maybe there will soon be a posting about Scala or JavaFX or both. Or something completely different.

Ciao