3rd generation NN, deep learning, deep belief nets and Restricted Boltzmann Machines

Highfreq · Nov 14, 2010

Re: Feature extraction

Krzysiaczek99 said:
Did anybody try to extract the features for Financial Time Series ??

According to this paper error from pruned feature set is lower.

There is also very good video lecture about it

http://videolectures.net/mmdss07_guyon_fsf/

Krzysztof

Feature extraction is a big word for data selection. But it is more involved than naively inputting price and some indis as most amateurs do. The later does not work. I more specifically look for features that are recurring and remove the junk. I also generate negative features that help the discrimination process. These ones alone make me a living. A good example is the shift of cycles to larger time-frames I touched upon before. The choice of technology (NNs, SVMs…) becomes secondary. My two cents.

Krzysiaczek99 · Nov 14, 2010

Re: Feature extraction

Highfreq said:
Feature extraction is a big word for data selection. But it is more involved than naively inputting price and some indis as most amateurs do. The later does not work. I more specifically look for features that are recurring and remove the junk. I also generate negative features that help the discrimination process. These ones alone make me a living. A good example is the shift of cycles to larger time-frames I touched upon before. The choice of technology (NNs, SVMs…) becomes secondary. My two cents.

I know that feature extraction is quite important, they were even competitions with money prizes in it on some test data sets.

What you mean by 'shift the cycles to larger TF ' ?? Larger TF is just less frequent sampling. Or you mean that the cycles can not be recognised on current TF any more but can be on higher TF so they are shifting to higher TF ??

Are you using any data mining tools like Rapid Miner or Weka ?? Matlab ??

Krzysztof

Krzysiaczek99 · Nov 15, 2010

For people not familiar with Weka here are the links

presentation of Weka and R

http://videolectures.net/bootcamp07_belanche_mldm/

and like 20 or more videos how to use it

http://sentimentmining.net/weka/

Highfreq · Nov 16, 2010

Re: Feature extraction

Krzysiaczek99 said:
I know that feature extraction is quite important, they were even competitions with money prizes in it on some test data sets.

What you mean by 'shift the cycles to larger TF ' ?? Larger TF is just less frequent sampling. Or you mean that the cycles can not be recognised on current TF any more but can be on higher TF so they are shifting to higher TF ??

Are you using any data mining tools like Rapid Miner or Weka ?? Matlab ??

Krzysztof

"Or you mean that the cycles can not be recognised on current TF any more but can be on higher TF so they are shifting to higher TF ??"

Exactly! The cycle shifts to the larger one when institutional traders place huge bets on the daily crossover. For example, if the 5min bar crosses the day open with just the right amplitude, then the orders shift on the day bar causing the current TF to loose equity. It’s time to switch TF.

I occasionally use Matlab and R. Mostly I use NeuroShell from Ward with a few plugins and write my own in C++. I don't know RapidMiner and Weka. They are certainly great tools to get accustomed to the new technologies but I fear they don't offer the level of customization that is often needed to make a generic technology work on market data. The solution often lies in the details.

Krzysiaczek99 · Nov 17, 2010

Weka

Yes those tools are very powerful very good for making comparative analysis, most algorithms and methods is already developed either for Weka or for Matlab so in my opinion they must be used. See this. I applied data from TradeFX for diffeent algos, than tried Adaboost to all those algos, than made feature extraction and did it again to see if there are any improvements, sadly no. All it took 1 hour work.....

Any info about Learn++

Tester: weka.experiment.PairedCorrectedTTester
Analysing: Percent_correct
Datasets: 2
Resultsets: 18
Confidence: 0.05 (two tailed)
Sorted by: -
Date: 11/15/10 5:13 PM

Dataset (1) rules.Ze | (2) trees (3) trees (4) trees (5) trees (6) funct (7) funct (8) funct (9) bayes (10) meta (11) meta (12) meta (13) meta (14) meta (15) meta (16) meta (17) meta (18) meta
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ipip_buy (1) 48.42 | 50.97 51.00 51.52 51.55 51.23 49.58 48.42 51.90 48.42 50.65 51.29 51.39 51.74 51.61 52.45 48.32 51.71
ipip_sell (1) 51.03 | 51.74 52.19 51.39 51.71 51.71 51.81 48.97 51.90 51.03 51.68 52.10 51.16 52.16 52.16 51.42 51.03 51.90
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(v/ /*) | (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0)

Key:
(1) rules.ZeroR '' 48055541465867954
(2) trees.ADTree '-B 10 -E -3' -1532264837167690683
(3) trees.J48 '-C 0.25 -M 2' -217733168393644444
(4) trees.RandomForest '-I 10 -K 0 -S 1' 4216839470751428698
(5) trees.SimpleCart '-S 1 -M 2.0 -N 5 -C 1.0' 4154189200352566053
(6) functions.LibSVM '-S 0 -K 2 -D 3 -G 0.0 -R 0.0 -N 0.5 -M 40.0 -C 1.0 -E 0.0010 -P 0.1 -model \"C:\\\\Program Files\\\\Weka-3-7\"' 14172
(7) functions.VotedPerceptron '-I 1 -E 1.0 -S 1 -M 10000' -1072429260104568698
(8) functions.Winnow '-I 1 -A 2.0 -B 0.5 -H -1.0 -W 2.0 -S 1' 3543770107994321324
(9) bayes.NaiveBayes '' 5995231201785697655
(10) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W rules.ZeroR' -7378107808933117974
(11) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W trees.ADTree -- -B 10 -E -3' -7378107808933117974
(12) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W trees.J48 -- -C 0.25 -M 2' -7378107808933117974
(13) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W trees.RandomForest -- -I 10 -K 0 -S 1' -7378107808933117974
(14) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W trees.SimpleCart -- -S 1 -M 2.0 -N 5 -C 1.0' -7378107808933117974
(15) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W functions.LibSVM -- -S 0 -K 2 -D 3 -G 0.0 -R 0.0 -N 0.5 -M 40.0 -C 1.0 -E 0.0010 -P 0.1 -model \"C:\\\\Program Files\\\\Weka-3-7\"' -7378107808933117974
(16) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W functions.VotedPerceptron -- -I 1 -E 1.0 -S 1 -M 10000' -7378107808933117974
(17) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W functions.Winnow -- -I 1 -A 2.0 -B 0.5 -H -1.0 -W 2.0 -S 1' -7378107808933117974
(18) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W bayes.NaiveBayes --' -7378107808933117974

Tester: weka.experiment.PairedCorrectedTTester
Analysing: Percent_correct
Datasets: 2
Resultsets: 18
Confidence: 0.05 (two tailed)
Sorted by: -
Date: 11/15/10 7:52 PM

Dataset (1) rules.Ze | (2) trees (3) trees (4) trees (5) trees (6) funct (7) funct (8) funct (9) bayes (10) meta (11) meta (12) meta (13) meta (14) meta (15) meta (16) meta (17) meta (18) meta
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
'ipip_buy-weka.filters.un (1) 48.42 | 51.58 51.58 51.58 51.58 51.58 51.65 47.68 51.65 48.42 51.58 51.58 51.58 51.58 51.58 51.61 51.58 51.65
'ipip_sell-weka.filters.s (1) 51.03 | 51.71 51.71 51.71 51.71 51.71 51.71 48.97 51.71 51.03 51.71 51.71 51.71 51.71 51.71 51.71 48.29 51.71
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(v/ /*) | (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0) (0/2/0)

Key:
(1) rules.ZeroR '' 48055541465867954
(2) trees.ADTree '-B 10 -E -3' -1532264837167690683
(3) trees.J48 '-C 0.25 -M 2' -217733168393644444
(4) trees.RandomForest '-I 10 -K 0 -S 1' 4216839470751428698
(5) trees.SimpleCart '-S 1 -M 2.0 -N 5 -C 1.0' 4154189200352566053
(6) functions.LibSVM '-S 0 -K 2 -D 3 -G 0.0 -R 0.0 -N 0.5 -M 40.0 -C 1.0 -E 0.0010 -P 0.1 -model \"C:\\\\Program Files\\\\Weka-3-7\"' 14172
(7) functions.VotedPerceptron '-I 1 -E 1.0 -S 1 -M 10000' -1072429260104568698
(8) functions.Winnow '-I 1 -A 2.0 -B 0.5 -H -1.0 -W 2.0 -S 1' 3543770107994321324
(9) bayes.NaiveBayes '' 5995231201785697655
(10) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W rules.ZeroR' -7378107808933117974
(11) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W trees.ADTree -- -B 10 -E -3' -7378107808933117974
(12) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W trees.J48 -- -C 0.25 -M 2' -7378107808933117974
(13) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W trees.RandomForest -- -I 10 -K 0 -S 1' -7378107808933117974
(14) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W trees.SimpleCart -- -S 1 -M 2.0 -N 5 -C 1.0' -7378107808933117974
(15) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W functions.LibSVM -- -S 0 -K 2 -D 3 -G 0.0 -R 0.0 -N 0.5 -M 40.0 -C 1.0 -E 0.0010 -P 0.1 -model \"C:\\\\Program Files\\\\Weka-3-7\"' -7378107808933117974
(16) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W functions.VotedPerceptron -- -I 1 -E 1.0 -S 1 -M 10000' -7378107808933117974
(17) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W functions.Winnow -- -I 1 -A 2.0 -B 0.5 -H -1.0 -W 2.0 -S 1' -7378107808933117974
(18) meta.AdaBoostM1 '-P 100 -S 1 -I 10 -W bayes.NaiveBayes --' -7378107808933117974

barnix · Nov 19, 2010

You can see my work about the C++ tester for MT4:
https://sites.google.com/site/ctesterformt4/

Krzysiaczek99 · Nov 21, 2010

barnix said:
You can see my work about the C++ tester for MT4:
https://sites.google.com/site/ctesterformt4/

Welcome to the thread.

Are you using mostly classification or regression methods. Can you post some
results of your predictors in form of confusion matrix or ROC curves ??

Krzysztof

Krzysiaczek99 · Nov 24, 2010

ROC analysis

Here is great info about ROC curves and link to presentation from P. Flach

http://videolectures.net/ecml07_flach_pto/

xan023 · Nov 24, 2010

Hi, I figured something like that.
Boltzmann-Gibbson Selection

double temp = 0.001;

int gibson(double buy, double hold, double sell) {
double T=temp;
double x_buy=MathExp(buy/T);
double x_hold=MathExp(hold/T);
double x_sell=MathExp(sell/T)+0.01;
double y=x_buy+x_hold+x_sell;
double y_buy=X_buy/y;
double y_hold=X_hold/y;
double y_sell=X_sell/y;
double random=MathRand()/32767.0;

if (random>y_buy+y_hold) return (2);
if (random>y_hold) return (0);
return (1);
}

What do you think of reinforcement learning ?

Krzysiaczek99 · Nov 24, 2010

xan023 said:
Hi, I figured something like that.
Boltzmann-Gibbson Selection

double temp = 0.001;

int gibson(double buy, double hold, double sell) {
double T=temp;
double x_buy=MathExp(buy/T);
double x_hold=MathExp(hold/T);
double x_sell=MathExp(sell/T)+0.01;
double y=x_buy+x_hold+x_sell;
double y_buy=X_buy/y;
double y_hold=X_hold/y;
double y_sell=X_sell/y;
double random=MathRand()/32767.0;

if (random>y_buy+y_hold) return (2);
if (random>y_hold) return (0);
return (1);
}

What do you think of reinforcement learning ?

Hi,

Can you explain in more detail what is a relation of your post to this thread ??

Can you post any paper which would show applicability of Boltzmann-Gibbson Selection to the financial time series ??

Regarding reinforcement learning and online algorithms. It's plenty of them around
and they can be a solution for non stationarity of financial time series. But so far I didn't have time to look more deeper.

Krzysztof

xan023 · Nov 24, 2010

Let me explain I was a member of a closed forum that dealt with the use of neural networks to play the forex choice fell on reinforcement learning.
Therefore, the forum was closed no soft from that forum can not normally be published.
But I have considerable knowledge and I pomuc in the development of new concepts.
The fraction of the code comes from the test Q-learning for classification and recording in the tables.

ps.It really works after every restart in the tester results are improving and the system obtains a greater profit
Krzysiaczek 99 Please email your.

Krzysiaczek99 · Nov 24, 2010

Online algos

For people interested in online algorithms there is some info

Plugin for WEKA called MOA - Massive Online Analysis

http://moa.cs.waikato.ac.nz/

Matlab Toolbox with online algos collection

http://dogma.sourceforge.net/index.html#about

% Online classification algorithms.
% banditron_multi_train - Banditron
% k_alma2_train - Kernel Approximate Maximal Margin Algorithm, with the 2-norm
% k_forgetron_st_train - Kernel Forgetron, 'self-tuned' variant
% k_oisvm_train - Kernel Online Independent SVM
% k_om2_multi_train - Kernel Online Multi-class Multi-kernel Learning
% k_om2_mp_multi_train - Kernel Online Multi-class Multi-kernel Learning, multiple passes
% k_omcl_multi_train - Kernel Online Multi Cue Learning multiclass
% k_pa_train - Kernel Passive-Aggressive, PA-I and PA-II variants
% k_pa_multi_train - Kernel Passive-Aggressive multiclass, PA-I and PA-II variants
% k_perceptron_train - Kernel Perceptron/Random Budget Perceptron
% k_perceptron_multi_train - Kernel Perceptron/Random Budget Perceptron multiclass
% k_projectron_train - Kernel Projectron
% k_projectron2_train - Kernel Projectron++
% k_projectron2_multi_train - Kernel Projectron++ multiclass
% k_sop_train - Kernel Second-order Perceptron
% pa_train - Passive-Aggressive, PA-I and PA-II variants
% pa_multi_train - Passive-Aggressive multiclass, PA-I and PA-II variants
% perceptron_train - Perceptron
% pnorm_train - p-Norm
% sop_train - Second-order Perceptron
% sop_adapt_train - Kernel Second-order Perceptron, adaptive version
%
% Online optimization algorithms.
% k_pegasos_train - Kernel Pegasos
% k_obscure_train - Online-Batch Strongly Convex mUlti kErnel leaRning
% k_obscure_online_train - Online-Batch Strongly Convex mUlti kErnel leaRning - 1st phase
% k_obscure_batch_train - Online-Batch Strongly Convex mUlti kErnel leaRning - 2nd phase
%
% Selective sampling algorithms.
% k_sel_perc_train - Kernel Selective Perceptron
% k_sel_ada_perc_train - Kernel Selective Perceptron with Adaptive Sampling
% k_ssmd_train - Kernel Selective Sampling Mistake Driven
%
% Auxiliary functions.
% model_init - General inizializiation function
% model_predict - General prediction function
% model_mc_init - Inizializiation function for Multi Cue Learning
%
% Miscellaneous.
% compute_kernel - Calculate the kernel values
% shuffledata - Shuffle input and output data
% demo - Demo of many classification algorithms
% randnorm - Sample from multivariate normal

Krzysiaczek99 · Nov 28, 2010

cost sensitive RBM

http://code.google.com/p/matrbm/

here is a link to RBM and DBN MATLAB code and 2 papers with some info about it. Does anybody knows if cost sensitive learning applies to DBN and how to implement it ??

Blaiserboy · Jan 22, 2011

An amazing thread......!!

Krzysiaczek99 · Mar 4, 2011

Time Series tool box

http://eeganalysis.web.auth.gr/

http://eeganalysis.web.auth.gr/matshelpfiles/index.htm

For interested people here is a link to Time Series analysis tool box. Different techniques
can be applied and different measures taken to have deeper knowledge about our time series.

Krzysztof

Blaiserboy · Mar 5, 2011

Re: Time Series tool box

Krzysiaczek99 said:
http://eeganalysis.web.auth.gr/

http://eeganalysis.web.auth.gr/matshelpfiles/index.htm

For interested people here is a link to Time Series analysis tool box. Different techniques
can be applied and different measures taken to have deeper knowledge about our time series.

Krzysztof

Thanks for posting.!

ambitious · Mar 29, 2011

gents, I am just getting my head around NN and wondered if you could point me to a newbie FAQ..or should I start with systematic systems?

Krzysiaczek99 · Aug 14, 2011

Pybrain

Here is a link to AI library in Python which can be used with the system from this thread.
The data exchange between Matlab and python is very easy, scipy has savemat and loadmat commands to save is Matlab format.

http://pybrain.org/

Krzysztof

Krzysiaczek99 · Aug 31, 2011

Hmm

Two interesting pdfs describing using of HMM for FOREX and stock market trading.

Very interesting to see other approaches and used methodology

Krzysztof

rathcoole_exile · Aug 31, 2011

or, to summarise,

"this is all a crock of shyte"

3rd generation NN, deep learning, deep belief nets and Restricted Boltzmann Machines

Junior member

Well-known member

Well-known member

Junior member

Well-known member

Newbie

Well-known member

Well-known member

Attachments

Newbie

Well-known member

Newbie

Well-known member

Well-known member

Attachments

Well-known member

Well-known member

Well-known member

Newbie

Well-known member

Well-known member

Attachments

Guest Author