nfoldxval

PURPOSE ^

Runs n-fold cross validation with a given classifier.

SYNOPSIS ^

function CM=nfoldxval( data, IDX, func_netinit, netparams, types, disctypes, fname, show )

DESCRIPTION ^

 Runs n-fold cross validation with a given classifier.

 Given n separate labeled data sets, trains classifier using n-1 data sets, test on
 remaining one.  Average results over all n such runs.  Shows overal results in one big
 confusion matrix and displays the results.

 The classifier is passed in as a parameter.  For this to work the classifier must follow
 certain conventions.  The conventions are as follows:
   1) The following must initialize the model (nin is the dim of the data):
       model = func_netinit( nin, netparams{:} )
   2) The crated model must have point to 2 function for training and applying it:
       model.fun_train    and   model.fun_fwd
   3) For training the following will be called:
       net = fun_train( net, X, Y );
   4) For testing the following will be called:
       pred = fun_fwd( net, Xtest );
 The format for X is nxp where n is the number of data points and p is their dimension.
 The format for Y is nx1.  Example of a classifier is: func_netinit = @clf_lda

 Given data in a cell array format, it might be useful to string it out into a single
 array:
   IDX = cell2mat( permute( IDX, [2 1] ) );
   data = cell2mat( permute( data, [2 1] ) );

 Overall error can be calculated via:
   er = 1-sum(diag(CM))/sum(CM(:))
 Normalized confusion matrix can be calculated via:
   CMn = CM ./ repmat( sum(CM,2), [1 size(CM,2)] );

 For a simple, small dataset, can do the following to do leave one out classification:
   [n,p]=size(data); IDX=mat2cell(IDX,ones(1,n),1);  data=mat2cell(data,ones(1,n),p);

 INPUTS
   data        - cell array of (ni x p) arrays of ni samples of dim p
   IDX         - cell array of (ni x 1) arrays of ni labels
   func_netinit- classifier init (see above)
   netparams   - classifier parameters (see above)
   types       - [optional] cell array of string labels for types
   disctypes   - [optional] array of types we aren't interested in {eg: [1 4 5]}.
   fname       - [optional] specify a file to save CM to, as well as image
   show        - [optional] will display results in figure(show) 

 OUTPUTS
   CM          - confusion matrix

 EXAMPLE
   load clf_data;
   nfoldxval( data, IDX, @clf_lda,{'linear'}, [],[],[],1 );      % LDA
   nfoldxval( data, IDX, @clf_knn,{4},[],[],[],2 );              % 4 k nearest neighbor
   nfoldxval( data, IDX, @clf_svm,{'poly',2},[],[],[],3 );       % polynomial SVM
   nfoldxval( data, IDX, @clf_svm,{'rbf',2^-12},[],[],[],4 );    % rbf SVM
   nfoldxval( data, IDX, @clf_dectree,{},[],[],[],1 );           % decision tree
   % for multi-class data
   nfoldxval( data, IDX, @clf_ecoc,{@clf_svm,{'rbf',2^-12},nclasses},[],[],[],4 ); % ECOC

 DATESTAMP
   29-Sep-2005  2:00pm

 See also CLF_LDA, CLF_KNN, CLF_SVM, CLF_ECOC

CROSS-REFERENCE INFORMATION ^

This function calls: This function is called by:
Generated on Sun 02-Oct-2005 19:35:28 by m2html © 2003