I've never been a particularly normal person, and I guess my hobbies are a reflection of that. I'm a programmer by trade, but I've recently taken an interest in historical statistics in light of the current financial crisis. I'll say up front that stats were never my strong point, so feel free to tell me if you think I'm doing anything out of line here, but I think it's pretty straightforward.

Last month on a lark I went to the Dow Jones web site and downloaded the historical Dow Jones Industrial Averages by day. After some minor massaging of the formatting, I loaded them up in Postgres and started poking around looking for similar events to put the current financial "crisis" into perspective.

One interesting query I hit upon is this:

select stats.*
        , prev.*
        , case
                when stats.avg_value > prev.avg_value then 'UP'
                else 'DOWN'
            end as direction
        , (stats.avg_value - prev.avg_value)/stats.avg_value * 100 as pct_change
from (
        select securityid
                , extract(year from valuetime)::int as year
                , avg(value)::numeric(20,2) as avg_value
                , stddev(value)::numeric(20,2) as stddev
                , (stddev(value) / avg(value) * 100)::numeric(20,2) as pct_dev
        from exchange.history
        group by securityid, extract(year from valuetime)::int
) stats
        inner join
(
        select securityid
                , extract(year from valuetime)::int as year
                , avg(value)::numeric(20,2) as avg_value
                , stddev(value)::numeric(20,2) as stddev
                , (stddev(value) / avg(value) * 100)::numeric(20,2) as pct_dev
        from exchange.history
        group by securityid, extract(year from valuetime)::int
) prev
        on stats.year = prev.year + 1
where case when stats.avg_value > prev.avg_value then 'UP' else 'DOWN' end = 'DOWN'
order by (stats.avg_value - prev.avg_value)/stats.avg_value * 100 asc

That's probably not very interesting, except that you can see how I'm getting my numbers. The history table it's drawing on is a series of daily Dow closing values.

Some interesting points from that query:

1. 2008 was the 11th worst year since 1897 in terms of percentage decline from the previous year at 15% decline. Top ten years were, in order of suckiness:

1932
1931
1930
1938
1907
1921
1974
1903
1970
1900

2. Standard deviation of 2008 was 1558.46, versus 526.07 in 2007.
3. In terms of the ratio between stddev and avg value of the Dow, 2008 was 8th, with a standard deviation of 13.61% of average Dow value.

So in terms of overall suckage, 2008 ranks pretty high. But we knew that.

From that, you might be tempted to think the whole year was awful, but in reality, it simply wasn't. We had 4 bad months -- January, June, October and November. October was the worst of 2008, losing 21.1%, followed by June at -11.7% January at 6.9% and November at -6.7% changes.

That said, June was the "craziest" month with the largest swings in prices -- stddev was 2652.53 versus 684.27 for October and 496.80 for November.

In terms of history, October was the 4th worst decline (21.11%) since 1897, following:

November 1929 (-37.79%)
April 1932 (-29.98%)
December 1931 (-29.03%)

June comes in at 30th worst decline at 11.6%, January at 73rd worst monthly decline (6.9%), and November at 78th worst decline at 6.67%.

How this plays into history remains to be seen. However, 2008 was only about half of the percentage decline seen during the depression years. For 2008 to be analogous to the depression, we'd need to see about twice the yearly decline, sustained for several years. 2009-2011 will determine which direction the country goes. The Great Depression had these types of losses for 3 years, followed by more moderate yearly changes.

Several other big-loss years turned out to be anomalies, followed by normal years. One example is 1907. 1905 and 1906 were huge gainer years, climbing 32% and then 16% before crashing 24% in 1907. The very next year, the decline was 1.6% and then 19% growth in 1909, followed by normal good, bad, and "blah" years where nothing really significant happened on the Dow. There were even several banner years up until 1929. In almost every case, losing years were erased in fairly short order.

I guess the point is that the current "crisis" is not unprecedented by any means, and isn't even half as bad as several years in the past. We as a nation need to be very careful not to make fiscal policy and sacrifice civil liberties for the sake of recovering from events that have happened many times in our past. The market has booms and busts, but there's no need to panic and give up our values and way of life.

So it looks like the DOW is going to close between +300 and +400 today, closing way up for the second day running. In particular, Citigroup is a big winner among the financials, currently up over 60% now that government money is involved. (http://www.breitbart.com/article.php?id=D94LH3080&show_article=1)

It seems like the market likes the government hand to give the reach-around, but I wonder when that's going to begin to chafe?

I'm not sure when people started thinking that it was a good idea to spend our way out of a debt problem, borrowing more and devaluing the money even further in the process. The whole concept is ill-conceived and will only put off the inevitable.

Thing is there's always a bump for a day or two when the government sugar daddy comes to visit, but soon enough reality sets back in. Keep an eye on Citi. They've got some fundamental problems, and they'll come back to haunt them in a day or two.

I wonder when we're going to start letting dead and dying companies die? At a certain point, the dollar will be worth nothing, and these companies will be addicted to the government teet. What then? How many times has Citi been bailed out already? Three in recent memory (at least twice by the Middle East, and now by the Federal Government -- in addition to the general industry bailout). How many more times? When will they be able to stand on their own? How many more trillions will they suck in before giving up the ghost?

My guess? We'll keep dishing it out until there's nothing left:

http://www.politico.com/news/stories/1108/15914.html

http://www.reuters.com/article/marketsNews/idINN1339368420081114?rpc=44

Looks like we might be finally learning our lesson! (What are the odds?)

I mean, the bailout of the financial sector is going so well, and is so well overseen. Why not open up the checkbook some more? We've already spent $5 Trillion on this "crisis," so what's a few billion more?

The nice thing about this whole mess is that it only proves the point that more government is not the answer because government usually can't even get itself organized, let alone properly oversee anyone else. Add to the fact that their usual approach is "If you don't understand it, throw money at it," and you'll see how they're only making matters worse instead of better. These guys don't need money. They need Chapter 7.

I would hope that a further bailout would be a hard sell to the public, given the lack of promised controls on the existing one. After seeing how hard the media pushed for the financial bailout and the picture of Armageddon they painted, I have no doubt they could sell it again for the auto industry.

They painted those Republicans in Congress who voted against the bailout as stupid, short-sighted, obstructionist, and a hundred other things, and it turns out it was exactly the opposite. But then, who's going to point that out?

Lots of money has already been spent in panic mode, under the banner of "We Must Do Something!!" because at least then it doesn't look like we sat on our thumbs and watched the economy fall apart. Enough already. We get it. You're clueless. Stop making matters worse.

http://www.portfolio.com/news-markets/national-news/portfolio/2008/11/11/The-End-of-Wall-Streets-Boom

This article highlights one of the main parts of Wall Street that I thought I was the only one to notice: people are full of s*it. I don't mean simple bravado to cover an insecurity. People here actually believe their crap, try to make themselves seem like the most intelligent people in the world (literally), make everyone who doesn't play the game look like an idiot, and then look to blame someone else when their world falls apart. That last part is key. It's all about the blame game here. It's not "Fix it," or "avoid doing stupid things," it's "do what you want, but look for the escape so you don't bear the consequences." ("here" being Wall Street in general.)

That blame game is what allows idiots to stay on top, because it means they're never exposed.

Honestly, that's why I'm taking a perverse joy in watching these nincompoops squirm. Sucks for me and everyone whose livelihoods they've jeopardized, but I'm glad to finally see them exposed. I cheered when that guy in LA (IIRC) thanked all the "idiots" whose daddies paid for the Ivy League educations, and who were the suckers to take the other sides of his trades.

This is as much a purge of stupidity as it is a correction for the problems they've caused. That's one reason I was against the bailout: you can't prop up unqualified, short-sighted idiots. You've got to let them crash and burn. Same in the Auto industry. Let them die. They bought their plots of dirt years ago. Let them lie in it now.

The people who inherited fabulous prosperity and didn't know what made it prosperous have steered this country into a very bad place, and the only way to fix it is to expose them for the idiots and liars that they are so that their conceit and failure is chronicled and sealed. For the same reason I'm secretly glad Obama won. He's the opposite of what I want in almost every way, but he's also so unqualified for the position he's in that he can't help but screw up big time. When he does, I want to see all those people who would have killed for the guy squirm. I want to see the parrots who stood around talking about how "dumb" Sara Palin is, as if it's a bon mot the 80th time it's repeated. So let it die, let it fall down, and let the phonies be exposed. In the meantime, we'll need to deal with them in every level of government, the media, and as the heads of corporations.

Damn. I think I'm going to shoot for a spot on talk radio.

In the news this week:

The US government could be entering a bottomless pit of bailouts if it starts propping up failing companies outside the financial sector: (Hoorah for corporate welfare! Glad we're propping up our biggest, most irresponsible and short-sighted companies for another few months!)

http://www.cnbc.com/id/27662540/

Lobbyists swarm the U.S. Treasury for a helping of bailout pie (Guess that's what happens when you open up the chum bucket to a bunch of sharks!)

http://www.iht.com/articles/2008/11/12/business/12lobbying.php

Hard Times, But Big Wall Street Bonuses: (Glad they're making good use of their bailouts...)

http://www.cbsnews.com/stories/2008/11/12/earlyshow/main4595179.shtml

American Express seeking $3.5 billion as part of government bailout program: (Business plan is to loan irresponsibly, sell the risk to others, and then ask for $3.5bn of taxpayer money so they can do it again!)

http://biz.yahoo.com/ap/081112/american_express_ahead_of_the_bell.html?.v=1

Pelosi supports new help for ailing US automakers (So these guys want to sell low-quality cars that no one buys, disregard market trends, and then ask for a bailout. Join the club. Glad the Dems have their hands in our pockets as much as the Republicans did. Maybe they'll realize that if we wanted to give GM and Ford our money, we'd BUY ONE OF THEIR FREAKIN CARS!)

http://apnews.myway.com/article/20081112/D94DD29O0.html

http://www.bloomberg.com/apps/news?pid=20601087&sid=aT78uzonQ2P0&refer=worldwide

Who Got That $2 Trillion? Fed Won't Tell (At least the Fed is refusing to tell us where $2 TRILLION (yes, "Trillion" with a "T") of our money went! They're afraid it'll cause the stocks to decline. Sure, then they'll tell us that it's the "free market" that doesn't work, and we obviously need the Fed to regulate it more. Way to be accountable, Fed. Note that this and about $800Bn of other money was NOT requested nor approved by Congress. )

http://www.bloomberg.com/apps/news?pid=20601087&sid=aatlky_cH.tY&refer=worldwide

http://www.bloomberg.com/apps/news?pid=20601087&sid=au7CSC89tBYs&refer=home

http://harpers.org/archive/2008/11/hbc-90003834

US May Lose Its 'AAA' Rating: (all of this leading to the devaluation of the dollar and possible default by the US Government. You can only print so much money before it's not worth anything. Then how do you repay your debt?)

http://www.cnbc.com/id/27641538

Paulson backs off asset plan; crisis deals at risk (Apparently even Paulson realizes this is a fool's errand, even if it only means he's correcting course to yet another fool's errand. Which fool's errand, you ask? Surprise! Giving banks even more taxpayer money, of course! At least he's trying, bless his heart.)

http://biz.yahoo.com/rb/081112/business_us_financial6.html?.v=6

Meanwhile, the jobs continue to melt away, after we lost 284,000 jobs in September: ("Job losses in September actually totalled 284,000 - 78% more than the 159,000 losses the government guestimated last month. November losses stretched to 240,000 on top of that, brining the unemployment rate to 6.5% - the steepest since March 1994.")

http://blogs.barrons.com/stockstowatchtoday/2008/11/07/job-losses-swell-past-million-signalling-deepening-recession/

First-ever layoff at the Post Office will claim 40,000 jobs:

http://www.labornotes.org/node/1947

And Wall Street begins plans to collectively lay off yet ANOTHER 70,000, in addition to the 150,000 they've already lost:

http://www.cnbc.com/id/27639559

So the bailout won't be helping. It's already proven to be insufficient before it's even given out, and the sharks are circling, trying to get the choicest morsels. Even Paulson is reconsidering what to do with the money, although he's apparently more of a fool than I'd initially given him credit for. Instead of buying the securities as he promised the public, where the money could be recouped at a later time, he's now thinking he'll just give it to the banks instead. Way to go, guy.

In the meantime, deepening job losses and spiraling debt are dragging down the demand side of the S&D, which produces a glut on the supply side and eventual additional layoffs.

All of this likely leads to the ultimate collapse of the US dollar, and the rise of the Amero (http://en.wikipedia.org/wiki/Amero). One more step on the path to a world-wide monetary union and eventual new world order, which will be the Only Possible Solution To Save Us All:

http://www.bloomberg.com/apps/news?pid=20601068&sid=aIcEL0jjSK74&refer=home

http://eux.tv/article.aspx?articleId=15099

Yes, I realize this last part is highly speculative and borders on "conspiracy theory," but it's sounding more and more plausible every day. Where last year at this time, it would have been dismissed out of hand, now you actually have to entertain the idea, and could even see how it could happen.

There was recently an article describing Pirate Bay's wish to encrypt all traffic by default. After writing a comment about how to solve the privacy problem presented by application ports, I got to thinking: Why would I want to facilitate the distribution of illegal material by making it technologically more difficult for police to prove their cases?

The thing is that, right now in this article, we're talking about copyright violation and how Pirate Bay wants to help its users avoid getting sued. However, the term "illegal" applies to much more than just copyright violation or kiddie porn.

I think my doubt came out of the idea that legality is somehow confused with morality -- an idea that must be summarily rejected on principle. After all, here in the US, we have laws that go against certain religious principles. An example would be anti-polygamy laws and polygamist sects of major religions. Polygamy, for those people and people in other nations of the world is a perfectly acceptable and harmless part of society. So just in this one example, we have a case where the morality of a very large number of people would be in opposition to a certain set of laws.

Another huge example has to do with dictatorships and authoritarian governments who restrict the flow of ideas and information by making them illegal. Take for example the Great Firewall of China, which is used to restrict the flow of potentially subversive ideas into the country. Should the Chinese people be subject to these free speech restrictions? We in America and most of the West consider free speech as paramount in our cultures, but in China, free speech is illegal. We in the West would encourage Chinese to visit a blog where "subversive" democratic and capitalist ideas are discussed, but the Chinese government would probably arrest these people. So again, another disconnect between morality and legality. Whose morality? Whose legality? It depends on your perspective.

The point is that it's in the best interest of humanity to ensure that communication channels are as open as possible, regardless of "legality." Does this mean that some of them will be used for some sort of "universally immoral purposes"? Yeah, it'll always happen that way. However, the pros will always outweigh the cons when it comes to protecting the privacy of communication channels.

The inalienable right to free speech is absolutely required if you want to avoid the exercise of the right to bear arms. Oppressive governments making immoral laws should never be able to subvert the free exchange of ideas through use of threats to the person. To that end, technology and technologists must do their part to ensure that technology remains as neutral to "legality" and "morality" as possible.

For the police's part, they are still free to use other methods -- seizing specific computers, subpoenas of server logs, etc. However, the transport should never be subject to mass violation of privacy. It's too easy to spread a net over an entire population.

When I'm not working for someone full time, I run a small consulting business. To keep my records straight, I've been developing a books and records system in Postgres and PERL, with a dash of Ruby. It currently supports, to varying degrees of completeness:

• Bank account records for multiple accounts (works very well)
• Expense report tracking
• Customizable CSV generation from any table in the database using arbitrary SQL filters in the form of a quick-n-dirty extract script.
• A data loader that can load schedules of days worked, invoices, and bank transactions using a CSV file. This makes it easy to format everything in a spreadsheet and then just load it all into the system.
• Simple PDF invoice generation.
• Multiple concurrent contracts at daily or hourly rates.
• Multiple employees
• Running calculation of bank account balance by posting date
• Simple time clock web app that can be hit from mobile phones to "punch in" or "punch out". This page can be put on a public web server, and there's a script to download the punches from the server to the main database.
• Automatic categorization of income and expenses based on patterns. As bank statements are loaded, the description is scanned for patterns, and a category is assigned if the pattern is recognized. For example, if the description contains "Shell Oil", the category would be assigned to "Fuel Expenses".

It's not complete by any stretch, but what it does, it does pretty well. The core functionality, invoicing and tracking bank accounts and credit cards, it does very well. There's no user interface apart from scripts, and parts of it definitely need some TLC.

That said, I think it's a great start for someone who's running a services-based shop who wants to have a books and records system that is totally open and expandable to their needs. If you think you'd be interested in trying out some software along these lines, please let me know. I can put the code up on Google Code for you to try out.

I read an fairly entertaining article today that coincided with a line of thought I'd had just hours before. The relevant part of that page is as follows:

Classic Handshake

Ah, the defacto male greeting. Yes, females use it too, but it's usually JUST a greeting if done by or to a female. The male handshake, however, is rife with subtlety and subtext, depending on point of contact, grip, pressure, length of duration, and the stature and status of the two parties involved.

This could really be an entire article unto itself, and probably one day will be. But for now, I'll touch on some of the more prominent components of a male-male handshake and their associated meanings:

• Hand Placement: Deep-set hand placement, where the webbing between the thumb and forefinger is set as close as possible to the same on the other male's hand, indicates a general respect for people. It shows that one's father showed them how to give a proper handshake. The further back the hand is set when the grip is closed, the less respect you apparently have for the person you are shaking hands with (and for most people in general).
• Pressure: Pressure equals confidence. The more pressure applied to a grip, the more confience the male is presuming to communicate they possess. Not enough pressure, and you are seen as weak. Too much pressure, and you are seen as overbearing and boorish, overcompensating for something. A firm handshake, where pressure is applied by both parties so as to create an equilibrium, indicates confidence on both sides.

I don't know whether it's just the people I interact with, or whether it's the area in which I live, but it seems that the importance or significance of a handshake has been completely lost on an entire generation of people. Most handshakes now are limp-wristed, two-finger versions that give the impression that it was a waste of the other person's effort, and that I should be honored that they even bothered.

Maybe it's old fashioned of me at the wise old age of 28, but when I meet a man who can't give a proper handshake, I write them off almost immediately. I'm not talking about those screwed-up handshakes where somehow you don't connect properly, I'm talking about the ones where you can see the other person couldn't have cared less and put zero effort into a simple show of respect. It says a lot about a person, their upbringing, and their attitude on life.

There are exceptions, I guess, mainly to do with other cultures. I've noticed that Indians and Latinos don't seem to put the same stock into it. I don't know if their cultures place the same significance into the gesture, or if it's even a part of their culture at all or simply something they learned in dealing with Americans. That said, any man who grew up in America should know and understand this simple gesture and what it says about him.

Anyway, I'd be interested to hear the opinions of other Slashdotters on this topic. Is it a dying gesture that shouldn't carry the weight I give it? Does it convey exactly what I think it does: a total lack of respect and effort on the part of younger people? You tell me.

I've never understood banks. I work in banks, I'm a customer of banks, and I guess I have a fairly good understanding of the banking industry. Here's what I don't understand, though, with this mortgage crisis/banking crisis thing going on.

1. Banks do stupid things, like lend money to people who can't possibly pay it back (By the way, it's not just mortgages. See "credit cards for illegal immigrants." now that many of them are packing up and going home voluntarily or otherwise.)
2. Banks get surprised when they don't get paid back.
3. Banks start worrying about the quarterly report and lay off workers to compensate. Bonus points for doing it during the holiday season.
4. Banks raise interest rates on adjustable mortgages to try to compensate for their losses.
5. Laid-off workers now can't pay their mortgages. Go to 2.
6. Homeowners with massively higher-than-expected payments now can't pay them back, even if they weren't particularly "high risk" to begin with. Go to 2.

Maybe that's a little oversimplified, but seriously, WTF?

Is it just me, or are banks making a bad situation worse by pulling down people who were otherwise doing fine and making them part of the problem instead?

We've seen the ripple effects these bad mortgages have on the industry. Fixed income instruments like bonds and mortgage-backed securities are supposed to be the tree when you're going out on a limb. They've turned out to be a tree that's dead and rotting, and a lot of people find themselves out on that limb holding termite-ridden sawdust. The ripple effects happen when one party is legally (contractually) obligated to cover someone else's bad bets. Chances are in times like these that they will be obligated to cover a LOT of bad bets. If they're unable to pay, everyone starts losing money.

For the car analogy, think auto insurance. You fully insure your $80,000 car: liability, collision, comprehensive, act of God, the works. You go out driving and total it while doing 110 in a 35. You walk away without a scratch, thinking "Damn, I'm glad I'm fully insured." When you make your claim, you find out that, unbelievably, 25 million other careless, drunken bastards crashed their $80,000 car that very same day, and they all have the same insurance company. Your insurance company now needs to pay out $2,000,000,000,000 ($2 trillion) within the next month. They file for bankruptcy and refuse to pay.

You and 25 million others are now out $80,000 for that car, even though you thought you were paying to be covered against exactly this situation. Worse, because 25 million people are now out looking for that same $80,000 car, the price has gone up to $110,000. If you want to buy that car any time in the near future, you're out $30k more due to market forces. Some people are actually paying this premium, but to do so, they can no longer put in that heated pool they were planning. There's your ripple effect. The market for pools was affected by a bad car insurance company, AND the cost of fixing the original problem has also gone up.

To make matters worse, the banking industry is now selling significant stakes to China and the Arab countries (See Morgan Stanley, Citibank, Bear Stearns, Merrill Lynch, for example). Not only does this put the country's financial system at risk of foreign influence, it also means that larger percentages of profits will be flowing overseas to these countries. That means less money (investment, employment) here at home.

All of this has detrimental effects on the overall economy, and it causes itself to get worse. When will people who call the shots decide that they need to stop exacerbating these issues by looking short term? Laying people off by the thousands doesn't help medium or long term, nor does selling controlling stakes overseas. It only buys short-term solutions at the expense of creating future problems.

Some analysts are calling what's happening as the makings of another Great Depression, Dust Bowl style. Only time will tell if that's true. For now, it sounds a bit alarmist, but we haven't yet been treated to the sheer selfishness of modern Corporate America. If you think they're bad when times are good, or even just "okay", wait til you see 'em when the excrement really hits the ventilation device.

This has got to be one of the more insightful AC posts I've read in a long time.

http://slashdot.org/comments.pl?sid=299321&cid=20628891

While he was ok through the first half of the post, I really keyed in when he started talking about politics and the total apathy of GenX and GenY. He's right, really, and he's the first one to put to words what I've been noticing.

• All politicians are old. Even Obama is no spring chicken. He just looks it compared to the rest of Congress.
• Gen X/Y tends to despise politics. It's not only apathy, but total disdain for the political process.
• Most politicians do not represent me. I've never identified with Democrats. I used to be a Republican, but I'm totally disgusted with them. They don't represent me anymore, and I have very few other choices. I vote, but it's a protest vote thrown to Constitutional and Libertarian candidates if I can find them.

I think politics have gotten too political. I know that makes no sense at first, but what I mean is that politicians have stopped trying to solve actual problems and have begun just trying to stay in power and take as much money as possible while there. Dems and Reps don't work together or see any common ground -- even where there is some. It's Us vs. Them in all things. We've forgotten that Us and Them are all "Us". We're all Americans, and this internal bitch-fighting is getting ridiculous.

Certainly in a country as powerful and rich as this, there will never be a lack of people who want to take the reigns. But, when the time comes, how much experience will they have? What will be their motives? Will they have any track record at all when every baby boomer in Congress retires?

I'm just disgusted with the state of politics, and, being born in 1979 and technically an X'er, I can understand why many of gen X and Y just stay the hell out of it. It's gotten stupid, and we've got more interesting things to do. Unfortunately, that attitude only guarantees that we'll never be adequately represented in government, and that we'll always be run by our parents' generation until they've all gone on to their respective nursing homes.