The goal of our work was to beat the current forecast methods used by the Department of Water Resources for the April through July American River flow. The Department of Water Resources uses an aggregation of human judgement and linear regression equations for generating their forecasts. Given their methods they are surprisingly hard to beat!

We spent a few months trying different Machine Learning methods with little success. Many of the methods we tried resulted in forecasts that were significantly worse than the current forecasts, a few methods such as a properly trained neural network gave forecasts that were comparable to the current forecasts. Finally, I decided to use a Support Vector Machine (SVM) for producing forecasts, after testing a large combination of parameters the forecasts started being significantly better than the current ones.

The data we used for generating forecasts is available online @ https://github.com/bjwbell/California-Water-Runoff-Forecasting. The takeaway message is that we improved the forecast relative error from ~65% to ~48%. The below table shows the forecasts for the last 10 years.

The forecasts currently used by the Department of Water Resources produced relative errors of 63.82% and root relative squared errors of 69.15%. Using modern methods for SVM’s gave us an increase in relative accuracy of over 15%! This was a fantastic result and shows the large payoff in keeping up with the state of art for something as ordinary as river flow forecasting.

The Art of R Programming is a straight forward explanation of R for programmers who are reasonably familiar with programming in another language. Matloff makes no assumptions of expertise in C or algorithms and his explanations are succinct and easy to follow.

If you’re aren’t familiar with R, it is a statistical programming language, with some similarities to Matlab.

Rating 9/10

The big advantages of R are (1) it’s high level, (2) reasonably easy to read, (3) functional in nature, (4) simple syntax. If you’re familiar with Python, it has a similar feel. Compared to complex languages such as C++, Java, etc, R is a breadth of fresh air due to the lightness of its syntax. That said as a programming language Python is nicer. R has a few annoyances (for me at least) that make it less pleasant to write in than Python.
A couple of those are:

• Non-standard assignment operator e.g. to assign 5 to x in R we use “x <- 5" instead of the normal "x = 5" used in other languages. This is annoying because a significant amount of programming is doing assignments and a two character assignment operator is twice as much typing. Contrast this with Python which uses the plain "x = 5".
• Vector creation using “c(1,2,3,4)”. Vectors in R are similar to lists in Python, it would be more natural to add a little syntactic sugar and use “[1,2,3,4]” for vector creation i.e. the same syntax as Python and many other languages.

The real reason to use R are its statistical libraries, it’s very widely used for statistics and is the most pleasant environment to work in.

The areas Matloff covers are:

Much of the material is available online in tutorials such as John Cook’s, R Language For Programmers. The real gems are the chapters “Writing Fast R Code”, “Interfacing R to Other Languages”, and “Parallel R”. These chapters have great information that is not easily discoverable otherwise.

“The Art of R Programming” is a fun read, albeit somewhat specialized. If you need to do statistical work as a programmer I highly recommend buying it and spending an afternoon browsing it.

Photos and slides are posted here and the slides are also viewable below.

Rating: 10/10

Miran Lipovaca has done a fantastic job of writing a book aimed at beginning Haskell programmers. I like his very straightforward writing style of introducing each topic with the minimum of complexity.

As you probably already know the book is available online for free at http://learnyouahaskell.com so the question becomes why purchase the dead tree copy? To be honest I don’t recommend buying a dead tree copy if you have a Kindle.

To summarize there are three ways to read the book:

• Dead tree
• EBook
• Online

My preference in order is the e-book on the Kindle, the dead tree version, and finally the online version. For some reason I don’t like the formatting of the online version as much as either the dead tree version or the ebook. It’s partly because I have a harder time reading books on the computer than either on the Kindle or in paper back.
No Starch did a great job of printing “Learn You a Haskell for Great Good!” on quality paper and the binding is excellent. As with all previous No Starch books I’ve purchased the physical layout and ink quality of the book are both excellent.

The table of contents is:

• 1. Introduction
• 2. Starting Out
• 3. Types and Typeclasses
• 4. Syntax in Functions
• 5. Recursion
• 6. Higher Order Functions
• 7. Modules
• 8. Making Our Own Types and Typeclasses
• 9. Input and Output
• 10. Functionally Solving Problems
• 11. Functors, Applicative Functors and Monoids
• 12. A Fistful of Monads
• 13. For a Few Monads More
• 14. Zippers

Miran does an amazing job of keeping each chapter short and interspersing lighthearted comments. The table of contents should be self explanatory, as you can see he covers mostly just the fundamentals of Haskell and ends with Monads. There is the extra chapter on zippers which seems like an afterthought.

Comparing this book to Real World Haskell, they serve slightly different purposes. “Real World Haskell” assumes that you are already reasonably proficient as a programmer whereas this book introduces each topic without assuming much programming background knowledge. Also “Learn You a Haskell for Great Good!” is meant less as a reference text and more for reading from start to end. They are both fantastic books and I highly recommend reading both of them. “Learn You a Haskell for Great Good!” is a gentler introduction to Haskell and I recommend reading it first.

Most of the following tips work equally well for using Emacs with other SQL databases besides MS SQL Server.

How to connect to SQL Server using Emacs

First you need to connect to a database. The command for connecting to a SQL Server instance is “sql-ms”.
Once you’ve entered the connection information Emacs opens a SQL query buffer in which you can type SQL commands. After you type a SQL command don’t forgot to enter “go” to execute the command.

Helpful commands for retrieving schema information

1. select * from sys.tables
Display the list of database tables

2. select left(column_name, 35), left(data_type, 15), is_nullable from information_schema.columns where table_name=’tablename’
Display the column names and column data-types for the table, ‘tablename’.

3. select left(CONSTRAINT_NAME, 100), CONSTRAINT_TYPE FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS WHERE table_name=’tablename’
Display the constraints for table, ‘tablename’.

4. select left(@@servername, 25), left(DB_NAME(), 25)
Display the database server name and the database name.

Improving Data Display

To improve line wrapping execute “M-x toggle-truncate-lines”, this prevents the soft line wrapping that Emacs normally does and improves the display of long lines.

To make the display of wide columns (such as varchar or char(8000)) readable use select left(columnName, 25) from table, this truncates the column width to 25 characters.

Useful Shortcuts

Because sql-ms and the other sql providers inherit from comint you can use all the nice emacs command line shortcuts. I use “C-c-l” to list the command history, “M-p” and “M-n” to cycle through commands, and “sql-send-buffer” (C-c-C-b) to send a buffer to the *SQL* mode buffer, this is extremely useful if I’m trying to tweak a large query. Use sql-set-sqli-buffer to set the SQL interactive buffer that’s used by sql-send-buffer.

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

The Blog-Health-o-Meter™ reads Wow.

Crunchy numbers

The average container ship can carry about 4,500 containers. This blog was viewed about 18,000 times in 2010. If each view were a shipping container, your blog would have filled about 4 fully loaded ships.

In 2010, there were 16 new posts, growing the total archive of this blog to 74 posts.

The busiest day of the year was January 20th with 673 views. The most popular post that day was Writing A Compiler In Haskell (Compiler Series Part I).

Where did they come from?

The top referring sites in 2010 were reddit.com, en.wordpress.com, moodle.cs.ualberta.ca, blog.llvm.org, and mail.live.com.

Some visitors came searching, mostly for line segment intersection, writing a compiler in haskell, nginx asp.net, nginx mono, and mono nginx.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

Writing A Compiler In Haskell (Compiler Series Part I) January 2010
11 comments

Line Segment Intersection May 2009
6 comments

Having Fun with Happy (Compiler Series Part III) April 2010
4 comments

Configuring Mono and ASP.Net on Nginx February 2009
3 comments

Playing with Alex (Compiler Series Part II) January 2010
4 comments

• A Web interface to enable the IT administrator to remotely manage the Linux and the Windows servers in real time;

• The management interface enables both performance monitoring and system control.

• The Performance monitoring component collects, manipulates, and displays statistics of a server and its applications such as CPU, memory, disk, and network utilization.

• The system control component enables the IT administrator to remotely execute commands against the managed servers, e.g. login, shutdown, restart, basic file management, etc.
“
The below diagram shows the system outline:

The class was organized into three teams (1) Windows agent, (2) Linux agent and the managing server, (3) Web server.

I was the team lead for team (2). The timeline of the project was the beginning of September through the first week of December. Which gave us a little more than 3 months. Our professor Dr. Ouyang had us give a sequence of demos during the three months and then submit a working system at the end.

What We Did Right
As the team lead for my team I instituted use of github.com for revision control. Github.com was a great choice, it’s an amazing site. The source code is available at https://github.com/bjwbell/CSC239-Linux-Team.

The web team used Dropbox for source code collaboration. Dropbox was a reasonable choice but they had to track revisions of the software manually so they ended up with directory names like “Website V017″. The Windows team was a mess they didn’t have any centralized collaboration system until the end where google code was used, https://code.google.com/p/239windowsteam/.

Another thing we did right was that my teammate Vinit Azad did a great job of specifying the interface between the agents and the managing server. This made it much easier to split the work up and have individuals program to the interface.

What We Did Wrong

• We didn’t impose hard deadlines. This caused portions of the project to be pushed later and later. At the end this caused integration issues where we were working on the managing server and the webserver while we were trying to integrate them.
• I didn’t fully specify the interface between the managing server and the web server. This caused problems when we integrated the two. It would have been much easier for the website team if I had specified a complete interface to the managing server that they could have programed against. This would have made integration significantly easier.

Recommendations
I would recommend setting a weekly schedule upfront with very specific milestones for each week with each milestone a portion of the project grade. The milestones should not be verbal confirmation that the milestone goals have been finished. Instead each team would email the professor their code with instructions on how to run it. With the code the professor can independently decide what part of the milestone goals have been finished. It is important that the team leads are not the ones to determine if the milestone goals have been completed because they are not independent evaluators.

Conclusions
The results of our class project were mixed. I could have done a much better job with specifying the interface between the managing server and web server. I didn’t realize until the end the importance of first specifying the interface and then coding to the interface. Given the constraints that it was a class project –so most of the team members had only had about one day per week to work on it– I believe a well defined schedule is very important to determine if the teams are falling behind and if so in exactly what areas.