The Miso Project :: Dataset

Creating a Dataset

To begin working with your dataset, you first need to import your data. You can import either local data (in the form of a variable that happens to contain your data) or remote data (in the form of a url that we'll fetch from.)

Local Importing

Out of the box Dataset can take local data objects or remote urls and import data in almost any common format.

JSON (including jsonp)
CSV
TSV (Any delimiter is acceptable, including tabs)

There is also a growing library of custom data importers such as:

Google Spreadsheets

Importing from a local object array

If you have an array of json objects, you can easily convert them to a Dataset like so:

          var myData = [
            { state : "Alabama", population : 4802740 },
            { state : "Massachusetts", population : 6587536 }
          ];
          
          var ds = new Miso.Dataset({
            data : myData
          });
          
          ds.fetch({ success: function() {
            log("Column Names: ", ds.columnNames());
          }});

Importing from a local "strict" format object

If you happen to have your data preprocessed in what we call a "strict" format, you can speed up your import slightly by initializing your Dataset with the strict flag:

          var myData = {
            columns : [
              { name : "state", data : ["Alabama", "Massachusetts"] },
              { name : "population", data : [4802740, 6587536] }
            ]
          };
          
          var ds = new Miso.Dataset({
            data : myData,
            strict: true
          });
          
          ds.fetch({ success: function() {
            log("Column Names: ", ds.columnNames());
          }});

Importing from a local delimited string format

If for some reason you actually have all your data as a delimited string on the client side (which is an unlikely but possible,) you can import that into a dataset object too.

          var myData = "state,population\n"+
                 "Alabala,4802740\n" +
                 "Massachusetts,6587536";
          
          var ds = new Miso.Dataset({
            data : myData,
            delimiter : ","
          });
          
          ds.fetch({ success : function() {
            log(ds.columnNames());
            log(ds.column("state").data);
            log(ds.column("population").data);
          }});

Note You can also import remote delimited data by simply providing a url parameter instead of the data one.
Note You can specify any delimiter string, not just the comma.

Remote Importing

Most of the time, your data will live somewhere else and you'll want to fetch it via a url. All the above formats would work except you would need to replace your data property with with a url property like so:

var ds = new DS.Dataset({ url : "http://myserver.com/data/mydata.json" });

Google Spreadsheet Importing

If you have a published Google Spreadsheet that you would like to import data from, you can do so with the following format:

          var ds = new Miso.Dataset({
            importer: Miso.Importers.GoogleSpreadsheet,
            parser : Miso.Parsers.GoogleSpreadsheet,
            key : "0Asnl0xYK7V16dFpFVmZUUy1taXdFbUJGdGtVdFBXbFE",
            worksheet: "1"
          });
          
          ds.fetch({ 
            success: function() {
              log(ds.columnNames());
            },
            error : function() {
              log("Are you sure you are connected to the internet?");
            }
          });

The google spreadsheet importer is utilizing the format specified here: http://code.google.com/apis/gdata/samples/spreadsheet_sample.html

Remote Polling

If you are handling a live data feed, you can initialize your dataset to perform ajax-based polling at regular intervals to fetch your data. There are three different ways in which this data can be merged into your existing dataset:

Appended - All new rows will be appended to the end of the dataset. This is the default behavior.
Reset - All the current rows in the dataset will be thrown out and the new rows will be put into the dataset. To enable this, set resetOnFetch to true when initializing your dataset. This will fire a reset event on a syncable dataset.
Unique - By specifying a column on which the data is supposed to be unique, new incoming rows will only be added IF the value in that column is unique. To enable this, set uniqueAgainst to the column name you wish to check against. Note, this is an expensive operation!

        // We will only make 5 requests in this example
        var requests = 5, 
            madeRequests = 0;
        
        // query twitter for tweets containing the term "javascript"
        // and get 5 per page. Note that twitter uses jsonp requests, 
        // so we toggle that flag too.
        var ds = new Miso.Dataset({
          url : "http://search.twitter.com/search.json?q=javascript&rpp=5&callback=",
          interval : 1000,
          jsonp : true,
          // Because of the structure of tweets:
          // https://dev.twitter.com/docs/api/1/get/search
          // we actually extract the rows from the returned data first.
          extract : function(data) {
            return data.results;
          }
        });
        
        ds.fetch({
          success : function() {
            // track how many requests we've made
            madeRequests++;
            // If we reached our max
            if (madeRequests >= requests) {
              // stop the importer.
              this.importer.stop();
              // output our current collection of tweets
              log(ds.column("text").data);
            }
            // update the number of rows we now have.
            log(madeRequests, ds.length);
          }
        });

Custom Importers

You may have noticed how easy it is to set a custom importer and parser in the dataset constructor by specifying the importer and parser properties. The import system can also easily be extended for custom data formats and other APIs. See the "Write Your Own Importers & Parsers" section for more information.

Fetching Data

Regardless of how you initialized your dataset (locally or remotely), it needs to be fetched for the data to be available. To begin fetching your data, simply call .fetch() on it.

Note that if your data is remote, it is especially imperative that you don't attempt to access your dataset before that call is complete.

Data can be fetched in one of three ways:

Pass success/error callbacks:

          var ds = new Miso.Dataset({
            data : [
              { one : 12,  two : 40,  three : 40 },
              { one : 1,   two : 40,  three : 40 },
              { one : 102, two : 430, three : 20 }
            ]
          });
          ds.fetch({
            success : function() {
              // do things here after data successfully fetched.
              // note 'this' references the dataset.
              log(this.columnNames());
            },
            
            error : function() {
              // do things here in case your data fetch fails.
            }
          });

Using Deferreds

If you have more than one dataset you need to wait on, or you might be a fan of using deferreds, you can use them as follows:

          var ds1 = new Miso.Dataset({
            data : [
              { one : 12,  two : 40,  three : 40 },
              { one : 1,   two : 40,  three : 40 },
              { one : 102, two : 430, three : 20 }
            ]
          });
          var ds2 = new Miso.Dataset({
            data : [
              { col1 : 1,  col2 : 400,  col3 : 420 },
              { col1 : 4,  col2 : 50,   col3 : 4220 },
              { col1 : 22, col2 : 0,    col3 : 24430 }
            ]
          });
              
          _.when(ds1.fetch(), ds2.fetch()).then(function() {
            // do things when both datasets are fetched.
            // note 'this' is NOT set to the dataset here.
            log(ds1.columnNames(), ds2.columnNames());
          });

ready Callback

You can also pass the dataset a ready callback that will be executed once the data is ready to be manipulated. This still requires fetching but allows you to have dataset-specific callbacks vs a single success callback for multiple fetches, for example.

Note that the individual ready callbacks are executed first and then the fetch callback gets executed.

          var ds1 = new Miso.Dataset({
            data : [
              { one : 12,  two : 40,  three : 40 },
              { one : 1,   two : 40,  three : 40 },
              { one : 102, two : 430, three : 20 }
            ],
            ready : function() {
              // do something specific to this dataset here when it's
              // been fetched
              log(this.columnNames());
            }
          });
          var ds2 = new Miso.Dataset({
            data : [
              { col1 : 1,  col2 : 400,  col3 : 420 },
              { col1 : 4,  col2 : 50,   col3 : 4220 },
              { col1 : 22, col2 : 0,    col3 : 24430 }
            ],
            ready : function() {
              // do something specific to this dataset here when it's
              // been fetched
              log(this.length);
            }
          });
              
          _.when(ds1.fetch(), ds2.fetch()).then(function() {
            // do things when both datasets are fetched.
            // note 'this' is NOT set to the dataset here.
            log("Both Datasets Fetched!");
          });

Data Types

Built in Types:

Dataset supports the following prebuilt data types:

number
string
boolean
time

Dataset will attempt and detect the type of each column when the data is being fetched. This will be done by looking at the first few rows of the data.

Overriding Detected Types

Dataset will attempt to detect what the type of your data is. However, if any of your columns are of a time format, it's much more efficient for you to specify that directly as follows:

columns : [
  { name : 'columnName', 
    type : '' 
    … [any additional type required options here.] 
  }
]

Dataset will take care of the actual type coercion, making it trivial to convert strings to numbers or timestamps to `moment` objects. For example, coercing the timestamp column into a time column and the total column to a numeric type would look like so:

          var ds = new Miso.Dataset({
            data : [
              { one : 12,  two : 40,  three : "2012 04_20" },
              { one : 1,   two : 40,  three : "2011 03_10" },
              { one : 102, two : 430, three : "2010 10_30" }
            ],
            columns : [
              { name : 'three', type : 'time', format : 'YYYY MM_DD' }
            ]
          });
          ds.fetch({ success : function() {
            log(
              // assemble all the dates' toString results
              _.map(
                this.column('three').data, 
                function(date) {
                  return date.toString();
                }
              )
            );
          }});
          
          

Custom Types

The type system itself can be extended to add new types for your data. The current type set is defined in src/types.js.

To define a new type, the required signature is as follows:

            // all types live under the Miso.types namespace.
            // Beware of overriding existing types. 
            // Nothing is stopping you from doing that.
            Miso.types.yourCustomType = {
            
              // provide a name for your type.
              name : 'yourCustomTypeName',
            
              // provide a method to test any incoming value for whether it
              // fits within the scope of this type.
              // return true if value is of this type. False otherwise.
              test : function(value) {},
            
              // provide a way to compare two values. This will be used by
              // the sorting algorithm, so be sure to return the correct values:
              // -1 if value1 < value2
              // 1 if value1 > value2
              // 0 if they are equal.
              compare : function(value1, value2) {},
            
              // how would this value be represented numerically? For example
              // the numeric value of a timestamp is its unix millisecond value.
              // This is used by the various computational functions of columns
              // (like min/max.)
              numeric : function(value) {},
            
              // convert an incoming value into this specific type.
              // should return the value as would be represented by this type.
              coerce : function(value) {}
            };

For example, we might define a custom phone type like so:

            // dummy phone number class
            var PhoneNumber = function(areacode, number) {
              this.areacode = +areacode;
              this.number = +number;
              return this;
            };
            PhoneNumber.prototype.valueOf = function() {
              return this.areacode * 10000000 + this.number;
            };
            PhoneNumber.prototype.toString = function() {
              return this.areacode + " " + this.number;
            };
            
            // Phone type. 
            // Converts a phone type to a phone type
            Miso.types.phone = {  
              name : "phone",
              // match phone numbers of the format:
              // XXX-XXX-XXXX or XXX XXX XXXX or XXXXXXXXXX
              regexp : /^(\d{3})[-|\s]?(\d{3})[-|\s]?(\d{4})$/,
              
              // tests whether an input string is a phone number
              test : function(v) {
                if (typeof v === 'string' || this.regexp.test( v ) ) {
                  return true;
                } else {
                  return false;
                }
              },
            
              // takes a string and converts it to a Phone type
              coerce : function(v, options) {
                if (_.isNull(v)) {
                  return null;
                }
                var split = (options.regexp || this.regexp).exec(v);
                v = new PhoneNumber(split[1], split[2] + split[3]);
                return _.isNaN(v) ? null : v;
              },
            
              // compares two PhoneNumbers. First compares by area
              // code, and then by the phone number
              compare : function(n1, n2) {
                var n1Value = n1.valueOf(),
                    n2Value = n2.valueOf();
                if (n1Value < n2Value) { return -1; }
                if (n1Value > n2Value) { return  1; }
                return 0;
              },
            
              // returns the numeric value of this datatype.
              numeric : function(number) {
                return number.valueOf();
              }
            };
            
            var ds = new Miso.Dataset({
             data : [
                { phone : "413 555 5555" },
                { phone : "413-444-4444" },
                { phone : "999 555 5555" }
              ]
            });
            ds.fetch({
              success : function() {
                // do things here after data successfully fetched.
                // note 'this' references the dataset.
                log("Column type detected: " + this.column("phone").type);
                log("Am I a PhoneNumber? " + 
                  (this.column("phone").data[0] instanceof PhoneNumber)
                );
                log(this.column("phone").data[0].toString());
              }
            });

Accessing Data

Columns

Each column in the dataset is of a Miso.Column type. We shall reference it as column for simplicity's sake.

A column has the following properties:

name
type
data - the data array for this column.
_id - a unique id assigned to this column at parse time.

While you can access the data inside dataset by directly accessing the data property on a column, it is NOT recommended as this will not handle any event propagation. Use direct access sparingly. For more information on accessing rows, see the Rows section.

Getting all column names:

ds.columnNames();

Note this will never include the _id column as it is internal to the dataset implementation and you shouldn't be messing with it.

Getting a column by name:

ds.column(columnName);

This returns the actual column object. Note that because the order of columns is not guaranteed (or should matter,) the fetching of columns is always done by name.

Iterating over columns:

  ds.eachColumn(function(columnName, column, index) {
    // do what you need here.
  });

Rows

Since dataset stores all the data column-wise, sometimes you may want to access a "row" object more easily than by iterating through columns. Note that the row object is not a direct reference to your actual data row (as in, if you modify it, it won't actually trigger a change in your dataset.) To change your dataset, you need to use the `update` method.

Iterating over rows:

          var ds = new Miso.Dataset({
            data : [
              { one : 12,  two : 40},
              { one : 1,   two : 40},
              { one : 102, two : 430}
            ]
          });
          ds.fetch({ success : function() {
            this.each(function(row) {
              log(JSON.stringify(row));
            });
          }});

Note that each row has a unique identifier assiged to it called `_id` in a separate column. Do not attempt to change that value unless you're feeling destructive. That identifier is used for caching purposes and changing it may make your data inaccessible through the API.

Row by Position:

if you're trying to get the Nth row, you can do so as follows:

ds.rowByPosition(5);

Note, this will return a row object that will not be a direct reference to your data. This will be a copy.

Row by id:

If you're trying to get a row with a specific id, you can do so as follows:

ds.rowById(423);

Note, this will return a row object that will not be a direct reference to your data. This will be a copy.

Events

Dataset has a very rich event system that allows you to bind to a variety of events on your dataset. By default, this functionality is NOT ENABLED. This is because event bindings are created automatically in certain cases (see more about selection and filtering) and unless that functionality is needed, there's no reason to create the bindings.

To enable evented behavior, set the sync property to true when initializing your dataset.

var ds = new Miso.Dataset({
  data : [
    { one : 12,  two : 40,  three : 40 }
  ],
  sync : true
});

Default Events

Presently, dataset fires the following events:

Event	Fired When	Precedence
`add`	Fired when adding a row to the dataset by calling `.add`	Primary
`remove`	Fired when removing a row from the dataset by calling `.remove`	Primary
`update`	Fired when updating a row in the dataset by calling `.update`	Primary
`change`	Fired when calling `.add`, `.remove` or `.update`	Secondary
`sort`	Fired when a dataset has been sorted.	Primary
`reset`	Fired when a dataset has been reset	Primary

Any of the default events can be prevented by passing the { silent : true } flag. See the appropriate methods for further instructions.

Binding

To bind to an event, call bind like so:

ds.bind("add", callback);

Event Object

When any of the default events trigger (except for sort) an event object gets created and passed down to the callbacks. The event object is structured as follows:

An event is of type Miso.Event
It has a deltas property that contains all the deltas that were generated for this specific event.

Deltas

An event is comprised of one or more deltas. Each delta can represent a different operation, allowing a single event to actually represent many modifications.

Each delta can look as follows:

{
  // the set of attributes that changed
  changed : { } or value

  // the old values of those attributes
  old : { } or value
}

When a row is added, there will only be changed attributes.
When a row is removed, there will only be old attributes.
When a row is updated, there will be changed and old attributes.
In certain cases the values of changed and old may not be an object but rather a numeric value. More on that in the Computed Values section.

Detecting Delta Types:

You can always check what type of a delta you've recieved by calling any of the following helper methods:

Miso.Event.isRemove(delta);

Miso.Event.isAdd(delta);

Miso.Event.isUpdate(delta);

For example:

          var ds = new Miso.Dataset({
            data : [
              { one : 12,  two : 40,  three : 40 },
              { one : 10, two : 32, three : 2 }
            ],
            sync : true
          });
          
          _.when(ds.fetch()).then(function() {
            ds.bind("add", function(event) {
              log(JSON.stringify(event));
              log("Is Add?", Miso.Event.isAdd(event.deltas[0]));
              log("Is Remove?", Miso.Event.isRemove(event.deltas[0]));
              log("Is Update?", Miso.Event.isUpdate(event.deltas[0]));
            });
          
            ds.add({ one : 10, two : 3, three : 24 });
          
            ds.bind("remove", function(event) {
              // We will now have an event with two deltas!
              log(event.deltas);
              log("New Dataset Length", ds.length);
            });
          
            log("Pre Remove Dataset Length", ds.length);
            ds.remove(function(row) {
              return (row.one === 10);
            });
          });

Custom Events

You can trigger your own custom events on dataset by just calling trigger when needed like so:

ds.trigger("myCustomEvent", arguments...);

Sorting

All data modifications should happen through the following add, remove and update methods.

Adding a row:

To add a row to your dataset, use the add method.

ds.add(rowObject, options);

          var ds = new Miso.Dataset({
            data : [
              { one : 12,  two : 40,  three : 40 }
            ]
          }).fetch({
            success : function() {
              log("Row Count Before Add", this.length);
              
              this.add({
                one : 100,
                two : 100,
                three : 100
              });
          
              log("Row Count After Add", this.length);
            }
          });

Adding a row will trigger the following events in this order:

add
change

Pass { silent : true } as the optional options parametere to suppress events.

Removing a Row:

To remove a row from your dataset, use the remove method. There are two ways to remove rows:

By providing a row id:
```
ds.remove(rowId, options);
```
Note the row id is the unique _id identifier each row has.

By providing a filter function:

ds.remove(rowFilterFunction, options);

For example:

// remove all rows where the population is > 1000
ds.remove(function(row) {
  return (row.population > 1000);
});

All rows that pass the filter function will be removed.

Either remove call will trigger the following events in this order:

remove
change

Pass { silent : true } as the optional options parametere to suppress events.

Updating Rows:

To update a row, pass the )id of the row along with the changed attributes like so:

ds.update(rowId, changedAttributes, options);

For example:

ds.update(rowId, {
  col1 : newVal, col2 : newVal ...
});

A call to update will trigger the following events in this order:

update
change

Pass { silent : true } as the optional options parametere to suppress events.

Selection

Dataset makes it easy to select sections of your columns and rows based on either static or function based criteria. Creating a selection will return a subset of your data that will function and be queriable in the same way your original dataset is. We call this subset a View. A view is almost identical a dataset except it is immutable(you can continue selecting subsets but you can't modify the data.)

Columns:

You can create a subset containing only some of the original columns from your dataset like so:

ds.columns(["one", "two"]);

Note selecting a single column this way will create a new dataset-like interface. If you're trying to just get a reference to a specific column in your dataset, just call ds.column("one");

        // initialize a new dataset
        var ds = new Miso.Dataset({
          data: { columns : [ 
            { name : "one",   data : [10, 2, 3, 14, 3, 4] },
            { name : "two",   data : [4,  5, 6, 1,  1, 1] },
            { name : "three", data : [7,  8, 9, 1,  1, 1] } 
          ] },
          strict: true
        });
        
        _.when(ds.fetch()).then(function(){
          var subset = ds.columns(["one", "two"]);
          
          log(subset.columnNames());
        });

Rows:

A more likely selection is a subset of rows that matches a particular criteria. You can select a row subset like so:

ds.rows(filter);

        // initialize a new dataset
        var ds = new Miso.Dataset({
          data: { columns : [ 
            { name : "one",   data : [10, 2, 3, 14, 3, 4] },
            { name : "two",   data : [4,  5, 6, 1,  1, 1] },
            { name : "three", data : [7,  8, 9, 1,  1, 1] } 
          ] },
          strict: true
        });
        
        _.when(ds.fetch()).then(function(){
          // create a subset of rows where the values in
          // column one are divisible by 2.
          var subset = ds.rows(function(row) {
            return (row.one % 2 === 0);
          });
          
          log("Subset length", subset.length);
          log(subset.column("one").data);
          log(subset.column("two").data);
          log(subset.column("three").data);
        });

Syncing:

If you enabled syncing on your dataset by setting { sync : true } during instantiation, your views will automatically update to reflect changes in your data. For example:

          // initialize a new dataset
          var ds = new Miso.Dataset({
            data: { columns : [ 
              { name : "one",   data : [10, 2, 3, 14, 3, 4] },
              { name : "two",   data : [4,  5, 6, 1,  1, 1] },
              { name : "three", data : [7,  8, 9, 1,  1, 1] } 
            ] },
            strict: true,
            sync : true
          }), subset;
          
          _.when(ds.fetch()).then(function(){
            // create a subset of rows where the values in
            // column one are divisible by 2.
            subset = ds.rows(function(row) {
              return (row.one % 2 === 0);
            });
            
            // bind to the _subset_ add event. 
            subset.bind("add", function(event) {
              log(event);
              log(this.column("one").data);
              log(this.column("two").data);
              log(this.column("three").data);
            });
          
            // now add a row to the original dataset that still
            // passes the filter. Watch it propagate to the view!
            ds.add({
              one : 100, two : 100, three : 100
            });
          
            // try to add a row that doesn't pass the filter.
            // This row will NOT be added to the subset and the
            // subset add event won't trigger.
            ds.add({
              one : 101, two : 100, three : 100
            });
          
          });

Computed Values

A pretty common requirement is to actually compute some basic statistics about your data.Most of the time those calculations happen on all the values in a specific column or a collection of columns, which is part of why we arrange our data in a column-wise manner. We call a computation that results in a single value a Miso.Product.

There are several default computations built in:

Max

        // initialize a new dataset
        var ds = new Miso.Dataset({
          data: { columns : [ 
            { name : "one",   data : [10, 2, 3, 14, 3, 4] },
            { name : "two",   data : [40,  5, 6, 1,  1, 1] },
            { name : "three", data : [400,  5, 6, 1,  1, 1] }
          ] },
          strict: true
        });
        
        _.when(ds.fetch()).then(function(){  
          log("Max of one", ds.max("one"));
          log("Max of two", ds.max("two"));
          log("Max of one & two", ds.max(["one", "two"]));
          log("Max of all", ds.max());
        });

Note that the max can be computed on numeric columns but also time columns!

          // initialize a new dataset
          var ds = new Miso.Dataset({
            data: { columns : [ 
              { name : "one",   data : [10, 2, 3] },
              { name : "date",  data : ["2011 01 01", "2012 03 05", "2001 02 01"]}
            ] },
          
            columns : [
              { name : "date", type : "time", format : "YYYY MM DD" }
            ],
            strict: true
          });
          
          _.when(ds.fetch()).then(function(){  
            log("Max of one",  ds.max("one"));
            log("Max of date", ds.max("date").format("DD/MM/YYYY"));
          });

Min

        // initialize a new dataset
        var ds = new Miso.Dataset({
          data: { columns : [ 
            { name : "one",   data : [10, 2, 3, 14, 3, 4] },
            { name : "two",   data : [40,  5, 6, 1,  1, 1] },
            { name : "three", data : [400,  5, 6, 1,  -1, 1] }
          ] },
          strict: true
        });
        
        _.when(ds.fetch()).then(function(){  
          log("min of one", ds.min("one"));
          log("min of two", ds.min("two"));
          log("min of one & two", ds.min(["one", "two"]));
          log("min of all", ds.min());
        });

Sum

        // initialize a new dataset
        var ds = new Miso.Dataset({
          data: { columns : [ 
            { name : "one",   data : [10, 2, 3, 14, 3, 4] },
            { name : "two",   data : [40,  5, 6, 1,  1, 1] },
            { name : "three", data : [400,  5, 6, 1,  -1, 1] }
          ] },
          strict: true
        });
        
        _.when(ds.fetch()).then(function(){  
          log("sum of one", ds.sum("one"));
          log("sum of two", ds.sum("two"));
          log("sum of one & two", ds.sum(["one", "two"]));
          log("sum of all", ds.sum());
        });

Note you can't add up dates, so don't try that one.

Mean

        // initialize a new dataset
        var ds = new Miso.Dataset({
          data: { columns : [ 
            { name : "one",   data : [10, 2, 3] },
            { name : "date",  data : ["2011 01 01", "2012 03 05", "2001 02 01"]}
          ] },
        
          columns : [
            { name : "date", type : "time", format : "YYYY MM DD" }
          ],
          strict: true
        });
        
        _.when(ds.fetch()).then(function(){  
          log("mean of one",  ds.mean("one"));
          log("mean of date", ds.mean("date").format("DD/MM/YYYY"));
        });

Syncing

If you have enabled syncing on your dataset by setting { sync : true } during your dataset initialization, you can also subscribe to changes in your computed product.

A big change to note here is that if you plan to subscribe to a specific computation, it is no longer a simple result (like a number or a date.) It now becomse a Miso.Product object that you can retrieve the value from like so:

ds.max(["one", "two"]).val();

For example:

          // initialize a new dataset
          var ds = new Miso.Dataset({
            data: { columns : [ 
              { name : "one", data : [10, 2, 3] },
              { name : "two", data : [1, 20, 3] }
            ] },
            sync : true,
            strict: true
          });
          
          _.when(ds.fetch()).then(function(){  
          
            var max = ds.max();
            log("Max of all",  max.val());
            max.bind("change", function(event) {
              log("New Value", max.val());
            });
          
            ds.add({
              one : 100, two : 100
            });
          });

Adding your own

If you want to add your own computations to your dataset, take a look at src/products.js for some examples (like max and min.)

For example, if we wanted to implement a product that returned a random value from the dataset, you could do it like so:

          // initialize a new dataset
          var ds = new Miso.Dataset({
            data: { columns : [ 
              { name : "one", data : [10, 2, 3] },
              { name : "two", data : [1, 20, 3] }
            ] },
            strict: true
          });
          
          Miso.Dataset.prototype.random = function(columnNames) {
            
            // find all the columns in question
            var columns = [];
            _.each(columnNames, function(name) {
              columns.push(this.column(name));
            }, this);
          
            return this._calculated(columns, function(columns) {
              
              // return a function that when re-run over a set of
              // columns can recompute the value we're looking for
              return function() {
          
                // assemble all the data 
                // values into a single array temporarily
                var values = [];
                _.each(columns, function(column) {
                  values.push(column.data);
                }, this);
          
                // flatten the values
                values = _.flatten(values);
          
                // get a random value from the total array of values
                // we gathered.
                return values[Math.floor(Math.random() * values.length)];
              }
            }(columns));
          };
          
          ds.fetch({
            success : function() {
              log(this.random(["one", "two"]));    
            }
          });

Note that this form will also support an actual subscribable product if your dataset is syncable.

Derived Datasets

Another pretty common operation is to transform the dataset into another dataset according to a method of some kind, such as grouping rows according to some criteria. When a dataset undergoes a transoformation like that we call it a derivative. There are currently only two basic derivatives available in Dataset with the plan to add more as they are needed: groupBy and movingAverage.

GroupBy

A groupBy operation involves splitting the data into groups based on a specific column, applying a function to the rows in each group and combining the results into a single dataset.

For example, when grouping the following dataset by the "state" column:

state	value
MA	130000
MA	420
AZ	2900
AZ	4

The result of the call:

ds.groupBy("state", ["value"]);

state	value
MA	130420
AZ	2904

By default the groupBy will sum up the values in the rows, but you can pass any method as an options argument like so:

          var ds = new Miso.Dataset({
            data : {
              columns : [
                { 
                  name : "state",
                  type : "string",
                  data : ["AZ", "AZ", "AZ", "MA", "MA", "MA"]
                },
                {
                  name : "count",
                  type : "number",
                  data : [1,2,3,4,5,6]
                },
                {
                  name : "anothercount",
                  type : "number", 
                  data : [10,20,30,40,50,60]
                }
              ]
            },
            strict: true
          });
          
          _.when(ds.fetch()).then(function() {
            var gb = ds.groupBy("state", ["count", "anothercount"], {
              // multiply all values
              method : function(array) {
                return _.reduce(array, function(memo, num){ 
                  return memo * num; 
                }, 1);
              }
            });
          
            log(gb.column("state").data);
            log(gb.column("count").data);
            log(gb.column("anothercount").data);
          });

Moving Average

A moving average of size N is a new sequence that is computed by taking the mean (or any other method) of the subsequences of N terms. For example, taking the moving average with a window size of 3 of the following dataset:

key	value
A	130000
B	420
C	1000
D	200
E	2900
F	4

Like so:

ds.movingAverage("value");

Will result in the following table:

key	value	(explanation - NOT IN TABLE)
C	43806	(130000 + 420 + 1000)/3
D	540	(420 + 1000 + 200)/3
E	1366	(1000 + 200 + 2900)/3
F	1034	(200 + 2900 + 4)/3

Note that you can also specify multiple columns like so:

ds.movingAverage(["A", "B", "C"]);

And an alternate method like so:

ds.movingAverage(["A", "B", "C"], { method : _.sum });

Syncing Behavior

If you are creating a derived dataset from a dataset that is syncable, you can subscribe to derived dataset'schange event.

Because of the inherent nature of a derived dataset, even the smallest change in your original data can cause many changes in your derived dataset. At the moment, those changes are not encompased in a set of deltas. Instead, the derived dataset gets recomputed. This is an expensive operation, but it reduces the code complexity substantially. We are open to discussing a better way of handling this situation, but for now, this works.