my journal 2

[Yamato]

gee, finally

Finally I am home and away from the very stressful day.

At the end of the day, when I was about to leave, vito asked me if he could ask me a work question.

Now, if you ask me something like that, and in the morning your boss told you to not disturb me, it is yet another proof that he was not innocent.

So basically we all agree that he's the one who did this, but no one wants to say it because he'd have to be fired, that dick.

Furthermore, the questions he asked me were bull**** questions, only meant to test me and see how mad I was. And my message was: don't mess with my stuff, but I'll help you with work because I am still a fair person. But I'll defend myself if you mess with my stuff. Fair doesn't mean you can fool me. His message was "please be my friend" and "I am not mad for having been reported to the bosses" and "I am not mad because it was me and not the cleaning lady".

Since I had rehearsed this situation, i was ready to help him. My policy is this: I will evaluate your behaviour on a daily basis, and respond with stick/carrot depending on how you behaved.

The stick of this morning (via the bosses) was for flipping my scanner on Friday. The help I gave in answering your bull**** questions tonight was for your good behaviour today.

The idea is: you behave well, I treat you well, and viceversa. This of course only because I can't kill him, so I have to keep a reasonable relationship with him, and maximize profits and minimize losses (I can't make permanent enemies, or they'll add up in time).

Also, I talked to my aunt, to tell her how it went.

She said that probably his messing with my stuff was like a revenge for the fact that I didn't accept to be his friend on his terms: I refuse his camaraderie, touching, give me five, talking... all that crap. All his stupid talk... I refuse him, and it shows. I don't care what he says. I consider him stupid. I always treated him fairly though, despite considering him stupid: I took him to the restaurant, I helped him, I let him keep light, window and door as he wanted them for half of the day. I am a fair person. The problem arose when he started behaving unfairly.

I think I've won this one. Hopefully there will be no more surprises in the future, because otherwise he'll be going against a few bosses, not just me. You can't just have fun at my expense, being only an intern, and without facing any consequences.

I am used to this type of thing. I am intolerant, I find people who resist me, but since I am right, I basically end up having my way. Yes, it's not normal that I don't like to talk and don't like to joke around, but if I want to just work and be taken seriously, you can make fun of me behind my back all you want, but you've got to let me work, and - since we're at the bank - you can't mess with my stuff either.

I know on the other hand that if we were in highschool i would pay for it big time, because you always have to pay if you don't fit in and you are not willing to beat people up. I did neither and I had to pay for it: such as getting some stuff stolen and damaged, by the evil people and their helpers. If you're not willing to conform to the norm, you have to be ready to fight or find a place where you can be alone and be the norm. You know what I mean? The Harvard intellectual is not going to hang around football players, or he'll get screwed for being himself and viceversa. Basically it's the same for everyone, and we're all minorities depending on where we go, in terms of age, race, gender, religion, ideas... etcetera.

Well, on this journal I am not a minority at least. And anyone who disagrees with me gets banned from the journal, so this is actually the opposite of what I've experienced in life. Here I can be in control. But after all, despite small intervals, I've always managed to control the situation that surrounded me. To the extent that I managed to become a control freak. Yes, because if you can't control almost anything and you grow up with a brother... you become tolerant. If you instead are only child and have everything under control, you do the opposite and will always try to gain control of the few things that escape your control. At least that's my guess given my point of view of being an only child. For example, if all of a sudden I found myself in a place where 9 people out of 10 are like Vito, then I'd probably give up on the idea of educating them and I would put up with all their abuses, until they eventually stopped. And I would even go out of my way to be liked by the vito-chimps.

But that's exactly why I turned to trading, as a means to make money and to put distance between me and the vito-chimps.

 

[Yamato]

still watching this:
http://www.letmewatchthis.com/watch-1876-The-Holiday

Very pleasant movie. I feel like making it last by watching it slowly. It is a very positive movie, wherever you turn you find someone nice. Maybe that's why I like it so much.

Damn!

Just when everything seemed almost perfect, they ruined the movie with the ending. It starts sucking really badly at minute 1:54.

I rated it a 14 out of 20 (initially I thought it was a 12).

Double damn!

I had to remove it from my "movies worth watching" list, because of what happens at minute 2:03. It's so bad that I can't even finish this movie. It's just not believable at all, and it's not a story you are attached to, but it becomes apparent that it's something totally made up. It's not a real story any more: it just took them 10 minutes (from 1:54 to 2:03) to ruin what they did in two hours. Damn.

 

[Adamus]

Just to reply to your post about the cushion money management strategy whether it's being conservative or not, I thought I'd say more, although it's probably superfluous.

Being conservative is relative - compared to me, or the optimal f money management approach which is less conservative than me, your approach is conservative.

In turn you are right, compared to your considerations I am optimistic and optimal f is reckless. In fact I think optimal f is reckless too. It bases its leverage on the largest loss that has occurred historically and we know with almost religious conviction that our largest losses are in the future.

The author Ralph Vince defined the strategy in 1992 or perhaps earlier and although I've kept tabs on his subsequent publications, it appears he hasn't changed his approach. Perhaps because he's not a trader himself, he doesn't see the point, or perhaps he's so rigid he doesn't like the idea of corrupting his pure strategy with the idea of defining your own maximum loss out of thin air, using the historical maximum loss only as a starting point.

But the key difference between the optimal f approach and the cushion approach is that the optimal f approach considers only how great the largest loss might be on any one trade, and the cushion approach takes into account how big the largest drawdown might be.

The fixed fraction bet size that comes out of the optimal f equation means that on losing, bet size is then reduced for the next trade.

Also according to optimal f, it never assumes the system stopped working, it just reduces the leverage when it loses. With a forex broker who allows small trades or with a huge account, it becomes obvious what the point of fixed fractional trading is and additionally it ignores whether the system has died. I think that should be considered separately from money management.

For instance, I could apply the cushion approach as you do to flag up when the system failed based on the worst historical drawdown, but use optimal f to give you the leverage. I could also choose either to allocate a fixed sum of money to my system, or I could be more advanced and allocate a fixed percentage of the account to a system and not reduce the leverage when it loses

It would be a nice situation to be in but at the moment I have just the one system in operation so it's all theoretical.

 

[Yamato]

Answering as I read:

I would say "how big the largest drawdown has been" rather than "how big the largest drawdown might be".

I can't reduce bet size with futures - that's my problem with optimal f, now i remember this is why i discarded it. Yeah, i remember now... the kelly criterion, close to optimal f. Sure: it makes sense but only if you can reduce your investment, which you can't do with futures (and a small capital).

Yeah, I am sticking with the cushion approach. Which is not just to say that we need to cover the maximum drawdown of the system if we trade one and the maximum combined drawdown if we trade 2 or more. The cushion also says that you can't go below zero. It almost says that you're trading just with your profit.

It's saying basically that before you trade a system with a given drawdown, your previous profit has to equal that drawdown. So that if things go badly you only get back to zero. Of course as you make more and more money that should change and you should allow a smaller and smaller part of your profit to be eroded by drawdown. So it means you go slower and slower in your scaling up. Otherwise if your systems fail ten years from now you will then lose everything you've made.

 

[Yamato]

I just realized that we all kiss up to people one way or another and we all try to get their friendship, and get the friendship of as many people as possible. I noticed this because I do a crazy befriending thing.

So what makes me better than butt-kisser vito?

The only thing that makes me better is sincerity and fairness. Am i really superior to him in these? I hope so. Otherwise it means I am not better than him.

I also work harder. I am willing to sacrifice more. This makes me better.

I do things better. I know more. I am more intelligent. This is why I am better. I am not better as far as not kissing up, because from that point of view we're all the same. We all want to be liked. Or maybe just me and vito: we have this in common. I just don't mess with his stuff. And if he has to work, I let him work without interrupting him.

 

[Adamus]

....... Otherwise if your systems fail ten years from now you will then lose everything you've made.

That's assuming I have no other way of determining when the system is dead.

But even then, as you add more systems you take away the profit from the system to allocate to new systems.

And of course you take money out because you want to do something with it.

So it's not quite that bad.

 

[Yamato]

Yeah, it is complicated. Sometimes I don't understand what assumptions you're making and other times you don't understand mine. We can't go any further unless we make some practical examples. From now on, let's do that.

I'll start small because now I don't have the physical and mental energies. And because sometimes you're gone for days or write objections without reading my previous posts.

Trading system A:
1) find its max drawdown during the whole back-tested period: 2000
2) we need a cushion of... say 3000 to trade it, but we don't have a profit cushion, since we haven't been trading. So we put up some money that we're willing to lose and call it a "max loss cushion", whereby we stop trading if we reach that loss, plus the tolerance (of 1000).
3) we keep on trading the system until we either lose 3000 or until we make 5000, at which we find a new system to trade.

Trading system A and B:
1) find max combined drawdown of A plus B during the whole back-tested period: 3500.
2) we already have a profit of 5000 which will cover the potential expected max loss of 5000 (tolerance of 1500).
3) we keep on trading until we either lose 5000 and go back to zero or until we make another 5000, reach a total profit of 10 thousand, and find a third system to add.

Trading system A and B and C:
1) find max combined drawdown of A plus B plus C during the whole back-tested period: 5000.
2) we already have a profit of 10000 which will cover the potential expected max loss of 7000 (tolerance of 2000).
3) we keep on trading until we either lose 7000 and go back to 3000 or until we make another 5000, reach a total profit of 15000, and find a fourth system to add.

Now you won't reply for 3 days, forget or ignore some of the stuff I wrote, and produce more objections. And that, I do not forgive. But that aside, let me say that I swear on the souls of my grandchildren, that I will not be the one to break the peace that we have made here today.

 

[Yamato]

It's 6 am here and damn, I can't sleep anymore. After the vito showdown.

I've already started to get drunk with some Lambrusco i found. That always helps me to sleep.

I went to the other room and somehow my father is not there. Every time this happens i think he might have died.

"Life goes on without him" comes to my mind. This is an expression I learned from this song:

http://en.wikipedia.org/wiki/Just_a_Gigolo_(song)
http://en.wikipedia.org/wiki/Louis_Prima

That's right, I am thinking of the possible scenarios if that happened. The first thing that I would do is probably quit my job. Every time something really bad happens to me, I quit my job. Girlfriend leaves me, I quit my job. I get a tumor, surgery, recovery, still quit my job. The thing is... my father is always the one who got me the new job afterwards, so in this case it would be final (I'd be quitting my last job ever). I've never walked into an interview where i wasn't hired from the start. That's how it works here in Italy. Not that I didn't deserve the position, but probably I would not have gotten it. I am not that good at selling and promoting myself. I am insecure. My dad made me insecure, so I always found it ok if he got me jobs, after making me so insecure.

The wine is kicking in.

[...]

So, for many english words and expressions I use there's a song behind it, and what comes to my mind about the possible scenarios is another song. Because that's how I learned "the english" (Scarface): songs and movies. Of course besides living in the States for a few years.

So...

As we live a life of ease
Every one of us has all we need
Sky of blue and sea green
In our yellow submarine

So... that's how I picture it. My father would die. And then I'd quit my job and move to the island. My mother wouldn't say anything against it. She's soft (Scarface). It would be nice if I could do all this while my father is alive, and because I am being supported by the investors, via our agreement. But if that weren't the case, I could still do it, my mom would let me do it. Given the shock. She would support me.

So I am thinking of the yellow submarine scenario. A life away from worries and people, a "life of ease". Quit job, move to island, shocked by death of father, everyone ok with my choice. Especially, father not disapproving of it. Yeah.

[...]

Last glass of wine. Last paragraph i can write.

That's the way i see it. I always liked isolation and underwater. They're very related. I am not bothered by fish. Fish don't bother me. Being underwater, swimming, is a way to be antisocial. I've got to get into scuba diving.

Wine is kicking in.

This is the right time to go back to sleep.

Lambrusco sucks. It tends to make me puke. I better go to sleep right now before the effect fades and I have to drink more.

 

[Adamus]

Now you won't reply for 3 days, forget or ignore some of the stuff I wrote, and produce more objections. And that, I do not forgive. But that aside, let me say that I swear on the souls of my grandchildren, that I will not be the one to break the peace that we have made here today.

You must be presentient.

I just read your paragraph "Trading System A" (1), (2) and (3) five times and perhaps through some subconcious urge to miscomprehend, I was unable to work out how you work out the numbers.

You have 2000 max drawdown, right.

You have 1000 tolerance, right.

What about this 3000 figure? Is that just the preferred amount or is it based on the other numbers? And the 5000? How do you derive that from the other numbers.

Since you are devoid of forgiveness, I won't ask for it. But I forgive myself, and happily. Forgiveness is good. Like greed. I used to pray for things and never get them, but it doesn't work like that. You take what you want, and then you pray for forgiveness.

 

[Yamato]

Mmh... your objections are too many for me to handle, post after post, without getting upset. I should get you on skype probably and speak (if you understand my English, with an Italian accent). I'll try in the next few days. Unfortunately you still don't have what it takes to make the ignore list. I keep looking for reasons, but you don't qualify. There's no superficiality in your posts. You're quite obsessive actually. So, you're the only one left who can post here and I'll have to deal with your objections in the future.

Besides, today i am in a great mood. I just got my part-time schedule approved for next year. We removed another half an hour from my daily schedule and now I'll be able to leave at 3 PM. At this rate, by the end of the decade, I'll just show up at work for 30 minutes and then come home, at about 9:30 am. And my dad will be happy because I didn't quit my job.

 

[Yamato]

wow...

I am almost feeling uncomfortable by how little is going wrong in my life right now.

No physical illness. No enemies at work. Vito taken care of.

Part-time approved.

I feel like something bad is approaching. My father always trained me to think in negative terms, and always look for things that could go wrong. Never to be carefree.

Right now I almost feel like I am missing some problems, to worry about. I feel like the problems are out there but I am failing to identify them.

This is partly because I've quit discretionary trading, and I have planned that the next account will be handled by me and the investors with a small symbolic share, just enough to keep me from gambling with it. So that compulsive gambling problem of mine seems to be solved, too.

Now all I have to worry about is to keep serious at work, because the minute I relax with vito, it will all go wrong and he'll stop respecting me. If he feels like you're his cousin he treats you like crap. He has to feel that I am a nazi SS guarding the prison he is in, to merely act like a polite person who doesn't interrupt me every 5 minutes and who doesn't mess with my stuff when I am not there.

If he doesn't get hired he will probably set my computer on fire on his last day. I have to remember to be there throughout his last 24 hours at the bank.

Anyway, as long as I keep a very serious attitude, don't lower my guard, and don't reply to his questions with anything but "fine, thanks", vito is now a problem of the past.

I guess I can now focus on the midas system and on the book I was reading. First comes that book:

resuming from page 47,
TYPES OF OPTIMIZERS


lmpllcit Optimizers

A mouse cannot be used to click on a button that says “optimize.” There is no special
command to enter. In fact, there is no special software or even machine in
sight. Does this mean there is no optimizer? No. Even when there is no optimizer
apparent, and it seems as though no optimization is going on, there is. It is known
as implicit optimization and works as follows: The trader tests a set of rules based
upon some ideas regarding the market. Performance of the system is poor, and so
the trader reworks the ideas, modifies the system’s rules, and runs another simulation
Better performance is observed. The trader repeats this process a few
times, each time making changes based on what has been learned along the way.
Eventually, the trader builds a system worthy of being traded with real money.
Was this system an optimized one? Since no parameters were ever explicitly
adjusted and no rules were ever rearranged by the software, it appears as if the
trader has succeeded in creating an unoptimized system. However, more than one
solution from a set of many possible solutions was tested and the best solution was
selected for use in trading or further study. This means that the system was optimized
after all!

Good point. I bet they will suggest to solve this problem with the out-sample, which is something I only recently learned. Well, reading a little further, there is no solution offered yet:

Any form of problem solving in which more than one solution is
examined and the best is chosen constitutes de facto optimization. The trader has
a powerful brain that employed mental problem-solving algorithms, e.g., heuristically
guided trial-and-error ones, which are exceptionally potent optimizers.
This means that optimization is always present: optimizers are always at work.
There is no escape!

Brute Force Optimizers

A brute force optimizer searches for the best possible solution by systematically
testing all potential solutions, i.e., all definable combinations of rules, parameters,
or both. Because every possible combination must be tested, brute force optimization
can be very slow. Lack of speed becomes a serious issue as the number
of combinations to be examined grows. Consequently, brute force optimization is
subject to the law of “combinatorial explosion.” Just how slow is brute force optimization?
Consider a case where there are four parameters to optimize and where
each parameter can take on any of 50 values...

Well, I don't have this problem because, whereas I do use a brute force optimizer (tradestation's), I have few parameters and I optimize them one at a time. Also, I want to know if the system i have in mind works, and not if any combination of those parameters works. The final system has to be close to what I initially had in mind or it can even be the opposite. Besides... random combinations that work are no good and they are easy to spot and discard because the values close to them don't work, or they don't work across markets. In those cases I discard them.

Notwithstanding, the optimization required 125 tests, which took 3 minutes and 24 seconds to complete on 5 years of historical, end-of day data, using an Intel 486 machine running at 66 megahertz.

Hmm, according to the quote above and to this link...
http://en.wikipedia.org/wiki/Intel_80486

...this part of the book must have been written not in 2000 which I thought was the year of this book but much earlier. Somewhere around 1995.

http://www.amazon.com/Jeffrey-Owen-Katz/e/B001I9W6SG/ref=ntt_athr_dp_pel_pop_1

February 29, 2000

I guess they took that part from one of their articles for Technical Analysis of Stocks and Commodities:
http://www.google.it/search?hl=en&r...e:www.traders.com&aq=f&aqi=&aql=&oq=&gs_rfai=

As they said in the Preface:

Finally, we would like to point out that this book is a continuation and elaboration of a series of articles we published as Contributing Writers to Technical Analysis of Stocks and Commodities from 1996, onward.

Yeah, they must have written that part in 1996. I'll even find the article. Nope, could not find the spot.

Since optimization is a problem-solving search procedure, it frequently
results in surprising discoveries. The optimization performed on the dual movingaverage
crossover system was no exception to the rule. Conventional trading wisdom
says that “the trend is your friend.” However, having a second moving average
that is faster than the first, the most profitable solutions in Table 3. I trade against
the trend. These profitable countertrend solutions might not have been discovered
without the search performed by the optimization procedure.

True: I created some systems simply because optimization proved me wrong ("surprising discoveries" as they say). Systems that were the exact opposite of what I was trying to build. Then of course, once again, it has to be a valid principle, that works across markets and timeframes, and works with variations to it, and it is not just a lucky combination or I won't build a system like that.

Successful user-guided optimization calls for skill, domain knowledge, or
both, on the part of the person guiding the optimization process. Given adequate
skill and experience, not to mention a tractable problem, user-guided optimization
can be extremely efficient and dramatically faster than brute force methods. ‘Ibe
speed and efficiency derive from the addition of intelligence to the search process:
Zones with a high probability of paying off can be recognized and carefully examined,
while time-consuming investigations of regions unlikely to yield good
results can be avoided.

Yeah, this is what I've gotten good at: "user-guided optimization". I know my databases. I know how to create systems fast. Now, with the help of the out-sample and all the markets I bought (13 futures), I can also defend my systems from myself, and fight illusions. Out-sample and cross-market studies were a great improvement for my tests. Of course, also forward-testing (which is a form of out-sample) was a great improvement, my first great improvement. But even before all these improvements, i had learned to create profitable systems. And to automate them. I would say the profitable automated trading for me happened in 2008. Before that, I was wasting time with over-optimized systems with too many parameters. Or systems that were automated but not back-tested. I wasted from 2002 to 2007 with that faulty testing and automating. Almost 6 years, wasted.

For now I'll stop here:
Page 51, Genetic Optimizers

 

[Yamato]

page 51, Genetic Optimizers

Imagine something powerful enough to solve all the problems inherent in the creation of a human being.

Yes, that's me, the human being, and that is what I want. I want a genetic optimizer. That will kill all those who bother me. I am all for it then, if that's what it is.

That something surely represents the ultimate in problem solving and optimization. What is it? It is the familiar process of evolution.

Hmm, so I suppose this is not for sale, is it?

Genetic optimizers endeavor to harness some of that incredible problem-solving power through a crude simulation of the evolutionary process.

A "crude simulation"? Is it anything I will ever be able to understand? It sounds interesting but I am afraid of formulas...

In terms of overall performance and the variety of problems that may be solved, there is no general-purpose optimizer more powerful than a properly crafted genetic one.

Ok, then, where do I get one? Let's start from a simple software where I can plug in my problems and they will be solved.

Genetic optimizers are Stochastic optimizers in the sense that they take advantage of random chance in their operation. It may not seem believable that tossing dice can be a great way to solve problems, but, done correctly, it can be!

So are you saying that I can just get a die and toss it and that is my software? That's quite tiring, and isn't that like regular optimizer?

In addition to randomness, genetic optimizers employ selection and recombination. The clever integration of random chance, selection, and recombination is responsible for the genetic optimizer’s great power. A full discussion of genetic algorithms, which are the basis for genetic optimizers, appears in Part II.

How is this different from a brute-force optimizer. I don't like the term "genetic". It's confusing.

Genetic optimizers have many highly desirable characteristics. One such characteristic is speed, especially when faced with combinatorial explosion. A genetic optimizer can easily be many orders of magnitude faster than a brute force optimizer when there are a multiplicity of rules, or parameters that have many possible values, to manipulate.

Oh, ok, so you're saying that it is actually faster. How can it be faster?

This is because, like user-guided optimization, genetic optimization can focus on important regions of solution space while mostly ignoring blind alleys. In contrast to user-guided optimization, the benefit of a selective search is achieved without the need for human intervention.

So you are saying that the genetic optimizer is like an intelligent person who can "sense" where the profit is? How could it be? Then why isn't this used within the brute-force optimizer? Why don't they just integrate the two and stop using this bull**** term "genetic"?

Genetic optimizers can swiftly solve complex problems, and they are also more immune than other kinds of optimizers to the effects of local maxima in the fitness surface or, equivalently, local minima in the cost surface.

This is not too clear, but you may be referring to the risks of overoptimization. I don't understand what you are talking about.

Analytic methods are worst in that they almost always walk right to the top of the nearest hill or bottom
of the nearest valley, without regard to whether higher hills or lower valleys exist elsewhere. In contrast, a good genetic optimizer often locates the globally best solution-quite an impressive feat when accomplished for cantankerous fitness surfaces, such as those associated with matrices of neural connection weights.

Your language is still unclear, but it vaguely sounds like you're saying that the genetic mother ****er is good at detecting in what areas lies profit and it would focus on an area where there's a lot of good results and forget about areas where there's a peak but nothing else around it, which is what I do when I check my optimizations on tradestation: if the best performance is isolated, I prefer worse performances that are close to each other, and pick the performance in the middle of those. What do you say? Is this what you were referring to?

Another characteristic of genetic optimization is that it works well with fitness surfaces marked by discontinuities, flat regions, and other troublesome irregularities. Genetic optimization shares this characteristic with brute force, user-guided, annealing-based, and other nonanalytic optimization methods. Solutions that maximize such items as net profit, return on investment, the Sharpe Ratio, and others that define difficult, nonanalytic fitness landscapes can be found using a genetic optimizer.

I like this. I guess you're saying that a genetic optimizer is nothing but a software that replaces me in looking at all the characteristics of a system's performance. Now just give me the name of one of them, and I'll download it, once and for all.

Ok, I am going to stop reading here and find out about this thing. I will look more on the web for them genetic optimizers. And see if I can download one, maybe on emule. This sounds like the genetic optimizer will build the whole system for me.

 

[Yamato]

nah... screw this

I already had started my search for a software that has a genetic optimizer, on elitetrader.com:
http://www.elitetrader.com/vb/showthread.php?s=&threadid=139504&highlight=genetic+optimizer

Also have found this:
http://www.genotic.net/tsgenotic.htm

But I am now realizing that I can't do it all at once. Trying genetic optimizers means a new platform and that is a whole new world. I can't do this right now. Better to do less and achieve it, rather than get too much in my hands and not achieve it.

So, screw genetic optimizers.

It's good enough that I've learned more about them and that I've gotten rid of my negative prejudice towards them. To me "genetic optimizer" sounded like "magic optimizer" until now, and it meant "bull****".

So I'll just keep reading the book. I might even screw the midas stuff ultimately and just focus on the book. One cannot do everything. One needs to focus. I am good at that. I am good at accomplishing what I start and not starting what I can't finish.

 

[Yamato]

resuming from page 52

This is where I had stopped reading:

Genetic optimizers shine with difficult fitness...

Yeah, it really looks like I will have to give them a try sooner or later:

Overall, genetic optimizers are the optimizers of choice when there are many
parameters or rules to adapt, when a global solution is desired, or when arbitrarily
complex (and not necessarily differentiable or continuous) fitness or cost functions
must be handled. Although special-purpose optimizers can outperform genetic optimizers
on specific kinds of problems, for general-purpose optimization, genetic
optimizers are among the most powerful tools available.

Damn, it's a pity that I can't give these things a chance:

Comparison of the brute force and genetic optimization results (Tables 3- 1 and 3-2,
respectively) reveals that the genetic optimizer isolated a solution with a greater net
profit ($172,725) than did the brute force optimizer ($145,125). This is no surprise
since a larger solution space, not decimated by increments, was explored. The surprise
is that the better solution was found so quickly, despite the handicap of a prematurely
stopped evolutionary process. Results like these demonstrate the incredible
effectiveness of genetic optimization.

... but their formulas are really way out of my reach, too complicated for me.

Page 54, Optimization by Simulated Annealing

Since genetic optimizers perform so well, we have experienced little need to
explore optimizers based on simulated annealing. In addition, there have been a
few reports suggesting that, in many cases, annealing algorithms do not perform
as well as genetic algorithms. Because of these reasons, we have not provided
examples of simulated annealing and have little more to say about the method.

Page 55, Analytic Optimizers

Analysis (as in “real analysis” or “complex analysis”) is an extension of classical
college calculus. Analytic optimizers involve the well-developed machinery of
analysis, specifically differential calculus and the study of analytic functions, in
the solution of practical problems. In some instances, analytic methods can yield
a direct (noniterative) solution to an optimization problem. This happens to be the
case for multiple regression, where solutions can be obtained with a few matrix
calculations. In multiple regression, the goal is to find a set of regression weights...

Wow, just too much for me. I don't understand this jargon... I failed math so many times, that's why it took me 8 years to graduate from college.

In other cases, iterative techniques must be used. The connection weights in a neural network, for example, cannot be directly determined. They must be estimated using an iterative procedure, such as back-propagation.

What the ****...?

I am going to quit this part, or skim through it really fast, because otherwise I'll just drop the whole book in discouragement.

Yup. Skimmed through it, and understood nothing at all of that paragraph. Let's put this horrible experience behind us. Anyway, it's pretty good so far, because out of 50 pages or so, I've understood 98% of them.

Page 56, Linear Programming
Still a bunch of words I don't understand, even here:
"linear programming"
"fitness functions"
"linear constraints"

Anyway here's how they end the paragraph:

Linear programming methods are rarely useful in the development of trading systems. They are mentioned here only to inform readers of their existence.

Oh yeah - that's very useful. Thanks. As if it weren't complex enough already. And now we're at 2 pages out of 50 that I didn't understand. 96% of intelligence and culture. 4% of studity/ignorance.

Page 57, HOW TO FAIL WITH OPTIMIZATION
Ok, I've now understood most of this book. It's totally useless to read it word by word. I need to get to the goddamn point. I can't waste going through the whole thing.

From now on there will be a lot of skimming and quoting the interesting parts. No more reasoning out loud here. I can't take 10 goddamn weeks to read this thing or I'll forget why I started reading it, which was to create new strategies - not to spend time reading.

Anyway, always better to read this book than to think and write about vito.

...annoying out-of-sample data sets have no place in the rose-colored world of the ardent loser...

One consequence of such a failure-laden outcome is the formation of a popular misconception: that all optimization is dangerous and to be feared.
In actual fact, optimizers are not dangerous and not all optimization should be feared. Only bad optimization is dangerous and frightening. Optimization of large parameter sets on small samples, without out-of-sample tests or inferential statistics, is simply a bad practice that invites unhappy results for a variety of reasons.

No, I know what I'll do - I have to write less. It will take me forever to read it if I keep coming here to comment on every sentence I read. I really could not finish it. I'll start reading more and commenting less. I will think the comments without writing them here. Anyway, by now you know this is a good book. So I'm sorry for depriving you of my comments and thoughts, but you can go on by yourself.

Gotta quote this though:
page 58:

Even when there are enough data points to avoid a totally artifact-determined solution, some part of the model fitness obtained through optimization will be of an artifact-determined nature, a by-product of the process.

59, all very good:

Large Parameter Sets
An excessive number of free parameters or rules will impact an optimization effort
in a manner similar to an insufficient number of data points. As the number of elements
undergoing optimization rises, a model’s ability to capitalize on idiosyncrasies
in the development sample increases along with the proportion of the
model’s fitness that can be attributed to mathematical artifact. The result of optimizing
a large number of variables-whether rules, parameters, or both-will be
a model that performs well on the development data, but poorly on out-of-sample
test data and in actual trading.
It is not the absolute number of free parameters that should be of concern,
but the number of parameters relative to the number of data points. The shrinkage
formula discussed in the context of small samples is also heuristically relevant
here: It illustrates how the relationship between the number of data points and the
number of parameters affects the outcome. When there are too many parameters,
given the number of data points, mathematical artifacts and capitalization on
chance (curve-fitting, in the bad sense) become reasons for failure.

That's good. I can't believe I've been so blessed to meet the investors who made me adopt the out-of-sample. They were so right. I can't believe i still created profitable systems without using it. Probably because I used very few parameters and very large samples.

No Verification
One of the better ways to get into trouble is by failing to verify model performance
using out-of-sample tests or inferential statistics. Without such tests, the spurious
solutions resulting from small samples and large parameter sets, not to mention
other less obvious causes, will go undetected. The trading system that appears to
be ideal on the development sample will be put “on-line,” and devastating losses
will follow. Developing systems without subjecting them to out-of-sample and statistical
tests is like flying blind, without a safety belt, in an uninspected aircraft.

On the other hand, out of all my previous 40 systems, 20 of them have failed in forward-testing (which is like an out-sample). If this ratio will be better with the new systems, created using the out-sample method, then I will confirm my faith in it.

60:

One way to accomplish this is with a data sample
that, even though not drawn from the future, embodies many characteristics
that might appear in future samples. Such a data sample should include bull and
bear markets, trending and nontrending periods, and even crashes. In addition, the
data in the sample should be as recent as possible so that it will reflect current patterns
of market behavior. This is what is meant by a representative sample.

All confirms my past work so far.

Finally, when running simulations and optimizations, pay attention to the
number of trades a system takes. Like large data samples, it is highly desirable that
simulations and tests involve numerous trades. Chance or artifact can easily be
responsible for any profits produced by a system that takes only a few trades,
regardless of the number of data points used in the test!

Yeah, of course.

An alternative that sometimes works is optimizing a trading model on a whole portfolio,
using the same rules and parameters across all markets-a technique used
extensively in this book.

Yup, done this, too.

So far I've learned little from this book, but at least I know I am on the right track.

 

[Yamato]

61, Verification of Results

To ensure success, verify any trading solution using out-of-sample tests or
inferential statistics, preferably both.

I am still wondering what inferential statistics are. It says it will tell me in the next chapter.

Some suggest checking a model for sensitivity to small changes in parameter
values. A model highly tolerant of such changes is more “robust” than a model
not as tolerant, it is said. Do not pay too much attention to these claims. In truth,
parameter tolerance cannot be relied upon as a gauge of model robustness. Many
extremely robust models are highly sensitive to the values of certain parameters.
The only true arbiters of system robustness are statistical and, especially, out-ofsample
tests.

Small changes in parameter values... interesting. The investors and I actually believe in that theory. But they are right as well: you can't make small changes in parameters that are very sensitive. But if they are, we can't call them "small". What does matter is that small insignificant changes do not cause a profitable system to become unprofitable. But this is all solved by the out-sample and and by testing the systems across several markets. On the other hand, before wasting the data-snooping there, it would be good to use our in-sample and maximize the system by verifying the small changes tolerance, as we believe. So the book is wrong to discard the "small changes tolerance" theory. It is a useful tool. But probably it's just semantics, and they call "inferential statistics" what we call "small changes tolerance".

 

[Yamato]

Gee... the chimp has been keeping quiet all day long and all week long. Practically a miracle. Obviously there was no tampering with my stuff either. But I can't lower my guard. It only works because I only answer "yes" and "no" to all his questions, with no improvisation and no originality. Basically I've erected a wall between us. He can't respect people he has a relationship with, so I'd rather not have any relationship with him. He perfectly fits the expression "he's a jerk": someone who can't be serious.

On the other hand, if he asks for help I treat him like everyone else and I help him, to teach him a lesson. In other words, this is the stick and the carrot. You try to get closer to me and I reject you, because you would quickly go back to disrespecting me. You keep it at a professional level, and I treat you fairly.

Needless to say, everyone (in my family) is praying that he won't get hired. But it's going to be very hard because this guy is good at manipulating people and he plays soccer with the boss, so he most likely has him in his pocket already.

 

[Yamato]

Every Day Above Ground Is A Good Day

Every Day Above Ground Is A Good Day

Another good day for me, because I won't have to go to jail for killing vito. Nor will I get fired for getting into a fight with him.

The training of the chimp goes on.

If he asks me for help, I help him.
If he starts being disrespectful, obnoxious, and a jerk, I ignore him.

That's me.

He's like this:

If I am nice and I take him out to lunch, he bothers me by talking to me (because he can't help it).
If I discipline him by not talking to him when he's obnoxious and because i need to work,
he (used to) mess with my stuff and flip my scanner.

Either way he bothers me. Actually he bothers me less if we're enemies than if we're friends.

So guess who's going to train who? He cannot train me to be nice to him, because he bothers me more when I am nice than when I am not.

So he actually did train me. He trained me to be cold to him and ignore him.

But, since I am fair, I can still interact with him by helping him (which is a way of being nice, carrot).

And by helping him, and using the carrot, I am hoping to shape his future behaviour: serious. If he's serious, he gets to interact with me. If he screws around, he gets ignored.

I'd say this is pretty good. But now it's the time he's bad-mouthing me. Today when he came back from lunch, I overheard him say that he was coming back to "prison" or something like that.

Which in many ways it's good, because all my bad reputation amounts to is being a prison guard, in the sense that in my room all you get to do is work and you don't get to screw around.

Not too bad to have this reputation when the guy complaining about me being a prison guard is a clown like vito.

 

[Yamato]

62, alternatives to traditional optimization

Self-adaptive systems work in a similar manner, except that the optimization
or adaptive process is part of the system, rather than the test environment...

Yeah, but then I already have these. If I use a range filter, it's like self-adaptive, isn't it?

At the very least, a trader venturing into self-adaptive systems
should have at hand genetic optimizer and trading simulator components that can
be easily embedded within a trading model. Adaptive systems will be demonstrated
in later chapters, showing how this technique works in practice.

Ok, then I guess these are not what I have. All this complex stuff makes it sound to me like it is not worth it.

There is no doubt that walk-forward optimization and adaptive systems will
become more popular over time as the markets become more efficient and difficult
to trade, and as commercial software packages become available that place
these techniques within reach of the average trader.

Huh? I hope not, or else it means that I'll be screwed, because I entirely rely on self-made simple and basic systems.

63, OPTIMIZER TOOLS AND INFORMATION

Aerodynamics, electronics, chemistry, biochemistry, planning, and business are
just a few of the fields in which optimization plays a role. Because optimization is
of interest to so many problem-solving areas, research goes on everywhere, information
is abundant, and optimization tools proliferate. Where can this information
be found? What tools and products are available?

Only quoting the interesting parts as usual.

Although sometimes appearing as built-in tools in specialized programs,
genetic optimizers are more often distributed in the form of class libraries or software
components, add-ons to various application packages, or stand-alone research instruments.

So, basically this only leaves out "built-in tools" in tradestation. They come in every other form. This is a pity because I don't even have the energy for installing add-ons.
Ok, this is the bottom line once again:

While genetic optimizers can occasionally be found in modeling tools
for chemists and in other specialized products, they do not yet form a native part of
popular software packages designed for traders.

Ok, good, I am almost done with this chapter, and getting to chapter 4. So far there's a 10% of stuff that I've read and not understood.

64, WHICH OPTIMIZER IS FOR YOU?

For many problems, brute force or user-guided optimization is the best approach.

Armed with both brute force and genetic optimizers, you will be able to solve virtually any problem imaginable.

Wow... I better do something about the genetic optimizer, sooner or later I better check them out.

TradeStation users will probably want TS-Evolve from Ruggiero Associates.

http://premium.working-money.com/Pr...dbname=software\software&tablename=soft_quest
http://www.google.com/search?hl=en&...ggiero+Associates&aq=f&aqi=&aql=&oq=&gs_rfai=
http://www.tradestationzone.com/tszContent.aspx?PageID=123&CompanyID=59
http://www.tradersstudio.com/
http://presentations.inxpo.com/Shows/Summit/05-08/Website/bio-murray.html

Excel users can try out the built-in Solver tool.

http://office.microsoft.com/en-us/excel-help/load-the-solver-add-in-HP001127725.aspx

Ok, since I could not find TS-Evolve on emule, I will commit to learning everything about excel's Solver before I even dream of moving on to chapter 4.

 

[Yamato]

Solver, excel's genetic optimizer

http://office.microsoft.com/en-us/excel-help/load-the-solver-add-in-HP001127725.aspx
http://office.microsoft.com/en-us/excel-help/about-solver-HP005198368.aspx?CTT=3

Solver is part of a suite of commands sometimes called what-if analysis (what-if analysis: A process of changing the values in cells to see how those changes affect the outcome of formulas on the worksheet. For example, varying the interest rate that is used in an amortization table to determine the amount of the payments.) tools. With Solver, you can find an optimal value for a formula (formula: A sequence of values, cell references, names, functions, or operators in a cell that together produce a new value. A formula always begins with an equal sign (=).) in one cell — called the target cell — on a worksheet. Solver works with a group of cells that are related, either directly or indirectly, to the formula in the target cell. Solver adjusts the values in the changing cells you specify — called the adjustable cells — to produce the result you specify from the target cell formula. You can apply constraints (constraints: The limitations placed on a Solver problem. You can apply constraints to adjustable cells, the target cell, or other cells that are directly or indirectly related to the target cell.) to restrict the values Solver can use in the model, and the constraints can refer to other cells that affect the target cell formula.

Use Solver to determine the maximum or minimum value of one cell by changing other cells — for example, you can change the amount of your projected advertising budget and see the effect on your projected profit amount.

Wow, just great. I am now using my first genetic optimizer. A good first simple step. Reading this book was a good idea. Even if it were just for this little thing.

What a ****ing idiot. I can't believe i am still thinking of that idiot, man.

Anyway, I am gonna make myself do this.

I am now looking at this file:

Microsoft Excel includes a workbook, Solvsamp.xls in the Office\Samples folder, that demonstrates the types of problems you can solve.

These are the two sample sheets I should be looking at:
"Maximizing Income" and "Portfolio of Securities".

I was looking on the web for excel solver and I found this stuff:
http://ddl.me.cmu.edu/ddwiki/index.php/Optimization#Optimization_Software

Then I also found that "palisade evolver" is better, an add-in for excel. I am downloading it now on emule.

http://www.palisade.com/evolver/

But this software suite includes evolver:
http://www.palisade.com/decisiontools_suite/

Here's what Katz says about palisade's evolver:

The Evolver product from Palisade Corporation is a good choice for Excel and Visual Basic users.

It's going to work. I will do it this way. I don't have much energies left for this stuff. I will limit my work to solver and to evolver. Then I'll have to move on to chapter 4.

I need to start from the simplest one: solver.

So I am now back to this:

These are the two sample sheets I should be looking at:
"Maximizing Income" and "Portfolio of Securities".

I will resume tomorrow.

 

[Yamato]

vito's daily behaviour towards me and best reaction

His behaviour and my reaction

1) kissing up: stop it with short neutral answers ("yeah...")
2) provoking: ignoring him
3) asking for help...
a) ...real: help him
b) ...kissing up (questions that he can answer by himself): help him
c) ...provoking (questions every 5 minutes): stop it with short neutral answers ("hmm..")


Frequency (in this past week) of behaviours and examples

1) kissing up: 4 times a day.
E.g.:
"do you want me to get you a sandwich?" "no,thanks"
"do you want me to close the door?" "however you want it"
"that guy is a fine person..." (speaking of my friend who left) "yeah"
"i'll do this orderly, like you taught me..." "hmm" (with indifference)

2) provoking: 4 times a day.
E.g.:
knocks on door when he comes back from bathroom. I ignore him.
walks towards my table but only to leave something on the table of the other colleague. I ignore him.

3) asking for help...
a) ...real: once a day
"can I ask you an excel question?" "yeah, sure" (I get up and go towards his table)

b) ...kissing up: once a day
"the printer is not working..." "yeah, later I'll go to see what happened" (then I don't do it)

c) ...provoking: never in last week
Previously, several times every hour:
"what's today's date?"
"how do you say this word in french?"
"how do you say this word in english?"
"what do you think of this?"
endless questions just to test my patience


Recapitulating behaviours and best reactions:

Kiss up: cut short with boring answers
Provoke: completely ignore
Ask for help: help if real or kissing up (I still need to reassure him I am willing to help), but ignore if he's provoking.

In general, simply be hostile to him, but turn nice if he unexpectedly behaves as a polite person.