I’m not going to take pictures of my room (sorry Kak Watiey heheh) – it’s such a sacred place for me. But they are very similar like these, except mine all in dark wood colour:

Ok that’s done and now last night I started looking at Django + PostgreSQL + Python in general. Was kinda slow, I have this habit of looking too far ahead (maybe it’s thinking too much) so I was worried that I would ruin my laptop in the future by blindly install everything inside. Typical woman? Being paranoid of nothing? Yea jumping in too early is not my thing, I’m scared if anything goes wrong. Laptop dies, application breaks, stuff happens yes I know but I don’t want it to be because I didn’t do my best to prevent it from happening.

Anyway, yea I’m embracing this temporary new found freedom – will put it to good use. Was so tempting to watch TV (I LOVE WATCHING TV – enough said) at nights but like my mom says “TV will be there forever waiting for you, go do something else better” so no worries there. Besides, I like programming. It’s like a drug. I remember Harish Pillay said during FOSS.my 2009, “How do you think an author wins an award for his/her book? He/She probably reads alot of other books to write it. So, how do you be good in programming? You read alot of codes.”

I love that man.

What’s data mining?

Well, basically it’s a process to get a knowledge (eg. if customers buy diapers, most likely they will buy beers too) from your data (imagine spreadsheets filled with numbers and strings). You may have thousands of records but how do you make sense of it? You mine those data, and get a pattern. From there you’ll get a knowledge. So from this you can put beers next to diapers in supermarkets to increase sales. That’s the general idea.

There’s a 90% chance that the data you got is not cleaned aka there’s missing values, some data are not consistent (eg. in sex can only have F or M, there’s a value Q?), and some data are just plain rubbish. The data need to go through preprocessing stage. For missing values, you have to filled them up, either using mean or median values, whichever is best for your data. The same goes to inconsistent data. This is where working with experts in the domain you’re working in is very important. You don’t want to remove what you think is rubbish but it actually means something.

Then we have data discretization process where you reduce that huge amount of data but they still carry the same value. Afterwards we normalize the data. After this is done, then the data is ready for the modelling exercise.

Classification

One of the benefit you get from data mining is classification. It’s where you can predict if you have this data (eg. male, married, doesn’t have children, have regularly purchase beer for the past year) whether beer purchase is likely. So it will classify to something like this purchase_beer equals to 0 (no) or 1 (yes). In order for the model to predict 0 or 1, the model itself has to be trained with alot of data. Train it until it reaches the accuracy we want, above 90% is good. A trained model with very high accuracy is going to be an asset as you can feed it data and it will spit out what we want.

The most difficult part in doing this for me is the data preprocessing part. You have to have quality cleaned data to produce quality results. You have to carefully select which kind of data is relevant to your goal (eg. would you want to include one’s job as one of the attributes considered for the diapers-beers example?) which is why having domain experts is important. They also have to determine how each data should affect the result (eg. job probably affects 10% but marital status affects 80% towards diapers-beers purchase).

I just love AI

There are just so many preprocessing methods & data mining hybrid techniques already been researched by academicians. I’m just so overwhelmed by the amount of technical papers on this. They probably not so much IT savvy like us, the implementors as they call it, but they definitely have the brain on that part of the world. We just have to scour through this massive database and get it to run on our apps. Well, should you need it that is. Coz processing thousands of data can take hours or days, some months, depending on your machine spec.

Open data

While doing data mining assignment last 2 weeks, I became frustrated with the unavailability of data in Malaysia. Sad. Maybe it’s still difficult for us to see it now, I’m already imagining all the stuff we can do with those data like in JPA, MOHR, MOH, MOHE – fuhhhh I’m all shuddery now. Even in OSCC, those training feedback forms and MyGOSSCON feedback forms – hmm whatever happened to those?

There are concerns of exposing private data I guess. Well, if you ask me make the data anonymous, as I couldn’t care less who got promoted last year. I only want the ’spec’ of that person who got promoted – age, sex, location, department, salary, is he respected by peers, does he drive, etc, that kind of stuff, you get the idea.

Hmm.. this will take another 10 years to realise, I think.

Semester I
The conditions of the class were pretty common and just above tolerable. No power points in sight except at the very front and the very back. When it was raining I prayed to god that I won’t die because of the cold temperature. Being in an IT faculty, taking an IT course, but no wifi? The ones they had was the prepaid ones. I couldn’t be bothered to search for any open shops in campus that’s open on Saturdays. Being one of the research university, free wifi should be available to all staff and students! Unless I need that rare publications, I can pretty much do anything on the internet.

Dr. Md Jan taught Data Structures (we got to know trees). He’s a considerate person. It’s a very straightforward paper, like Maths, that I loved it! He’s always open for discussions, jovial and committed. I like him because he made use of internet in his lectures. He’s current. He let us choose our own programming language for his assignments. Being an open source person, that’s freedom of choice and I liked it.

Prof. Patel taught Computer Network. I just couldn’t bring myself to like this paper. It felt like out of this world. He came across as someone who teaches just to pay the rent. Nothing’s wrong there of course, it just felt ordinary. I couldn’t understand his assignments, I did it just to pass the test. Network is something that I don’t wish to dwell in so much.

Semester 2
Hey the classroom looks so much better. New floor, new chairs so I the chilling problem gone. They set up power points! And there’s wifi! Applaudable effort for me even though it’s not stable all the time but the thought is there. Keep it up!

Dr. Salwani teaches Software Engineering. Her expertise is in AI but she used to be a developer so it’s actually captivating to listen to her lectures (at least for me coz I can relate). She used real life scenarios and real life problems & solutions. I think she’s well respected in her field to other academicians in Malaysia. She’s got a thing with open source that I’m not so comfortable with (from past experience maybe?). She’s tough and kinda strict but lacks a bit in managing assignments deadlines. We have 2 assignments still not sure of the deadlines. I think my classmates wish she would just take back the those 2 since other one we’ve been doing is quite huge!

Dr. Azura teaches Advanced Artificial Intelligent. She’s one of the top AI researchers along with Dr. Salwani, although her interest is in data mining. The work they’ve done, I mean, seriously, big corporations should hire/consult these 2 – they can give you patterns and predict things! I am astounded that I had not looked into AI before (I studied maths doing my first degree and I though AI is very physics-like haha thanks to tv shows), that I feel those CS graduates should all work in AI industry! Don’t just program a program, program an AI program. Anyway, she’s a great person, abit chatty in the classroom although I don’t mind it since the class is 3-5pm, and a very knowlegable person.

Conclusion?
Although the lecturers have been great, I’ve had some problems with my group assignments. I like if the assignments is individual coz I don’t have to rely on others to do my work. I have no problems with working in a group, it’s just frustrating some of them like to work last minute. So I usually did my part early and trust that they would take that and continue. When it’s time to submit, it’s disheartening looking at the end result so messy and unorganized. Oh well.

Being the only programmer in the group really sucks sometimes. The group did their part, I believe as much as they could but I get the right to grumble. Non-programming assignment, I contribute to the group. When it comes to programming, only me doing it? It’s because they have already forgetten how to program. This is real, so real that it sucks being a programmer in this case.

Studying really takes a toll in your life. Your priority changes abit. Not much if you’re single, alot if you’re married. I understand my group have husbands and children to care for but it’s not good if you abandon your team on not one, but many presentation days. My advice to married women who wants to pursue study:

[1] Let everyone in your family know your intention. If you’re studying, let them know you’re studying. Your priority has changed abit. Last time you go straight to the kitchen after work 5 times a week, now you only do it 2 times a week. There’s just some routine stuff you can’t do anymore.
[2] Your husband should understand what being a grad student means. It doesn’t mean you still can be with him everytime going to everywhere, not until you finish studying. It doesn’t mean he can continue old habit of not doing anything around the house after work (just an example). You wife has other stuff to do now, give a take a little now can we? If you can’t work together, I assure you, you’re going to be staying up late every night just to catch up with the lessons.
[3] Please understand the sacrifice you have to make if you’re studying. You’re going to be tired, busy all the time and you can’t afford to be at all family occasions/kenduri kawen kawan anymore. If you’re studying, you want to be good at it and because it means something to you. If not why do it at all?

Anyway, I actually enjoy the whole period so far although I don’t have time for anything else. You want it, you have to sacrifice your time. I now know some stuff that I probably wouldn’t have known if I’m learning on my own. Granted there are times I felt like quitting – but that’s just normal I guess.

Education is not cheap these days. I should squeeze everything out of it so I get the money worth. All in all I think UKM is an ok place for me. As long as they have good lecturers, that’s the most important thing and so far I think they have pretty good ones at FTSM.

Fuzzy reasoning is useful when you’re dealing with ‘high IQ’, ‘average IQ’, ‘low IQ’, etc. How would you translate ‘high’ or ‘average’ into crisp numbers like more than 160 it’s ‘high IQ’. If it’s more that 130cm it’s ‘average IQ’. Of course the numbers can’t be used for everyone (I may have a different opinion on how much is ‘high IQ’). And you can’t use simple algorithm if and else like we normally would do. Fuzzy sets on the other hand allows you to have 160 falls under ‘high IQ’ and ‘average IQ’ at say, 0.8 and 0.6 (on a scale 0 to 1) each. Once you have defined all that, you have to have your fuzzy rules to work on. Given someone’s IQ level and his test score, you can calculate his learning difficulty level and then maybe you can have this result to propose a new set of tests that match his level.

Artificial neural network emulates how the brain works, well sort of. It has the capability to learn and predict stuff. Like the H1N1 case, the model will learn to reach a very good predict whether the patient has high risk of being infected with H1N1 or not. It is much dependant on number of cases, the more cases you feed the model, the better the result will be.

Personally I think it’s all mathematics. Everywhere you see calculations. Once you understand how it works, it’s pretty easy (the data I worked on is not that much hahaha). In real world, the data set is huge, so data mining is actually intriguing, I think. Worth knowing.

In the risk of my classmates meniru, I’m going to upload what I have done so far, which I doubt that they’ll find it here until much later. If you manage to get it here anyway, hey I really don’t mind you looking around. Just be kind enough to give feedbacks there’s mistakes in the doc .

Okay. HUNGRY. AND SLEEPY.

Anyway, the AI course – I like the lecturer already, she encourages us to use open source in doing assignments. And she’ll tailored the assignments to fit the background of her students. One thing that I like is that she’ll fail anyone who doesn’t submit assignments even though it carries 40%.

And yeah the introduction to AI that we went through today has made me to look at MyMeeting on a whole new perspective. It’s a far fetched thing but I’d like it to be able to tell the user what will happen to his projects in advance, say 3 months early. It would be cool if the system can predict whether or not the action will be taken care of by specific agency or not. So this can evolve to something else, like picking the right agency to implement a task, since the system holds its past performance. Fuyyoh very high ambitions, feeling rather productive at this minute… hahaha.

You know I miss doing integration between systems. Too bad I’m working for OSCC which has a few products but I’m only working on one product. I reminisce the day when procurement department can approve parts purchase by the red light indicator beside its name, and how customer service department can send alerts automatically when reservoir level is too low. Initially there were many independent apps going on there but when it’s integrated, great things can happen.

From all these apps, the high level management then only can see, for example, how much they have spent, how much water has been saved, at what pace the water is being consumed, are they ready for the next quarter etc. This kind of thing helped alot in driving the direction the company is going. The point I’m trying to make is, sometimes big bosses they ask for something high level like this when the data are not there to begin with.

Hmmpphhh I’ve to go, next class is starting.

1. Read a file (*.txt) and determine frequency of each characters and total bits.
2. Build a Huffman tree based on previous frequency table. Each node keeps a character & its frequency, path (eg. 011, 0011, 01110 etc), vertex flag (whether it’s a left or right child) and visit flag (whether it has been visited or not – for node searching later). Note though, not every node keeps a character, only leaf nodes do (Huffman tree properties). Sorting algorithm to build priority queue to work with when building the tree is selection sorting algorithm.
3. Follow every root-to-leaf path to get the bits for each character. This script uses breadth-first search algorithm to look for nodes inside the tree.
4. Write compression information in a new file (compressed one) but this script only write the bits sequence into the file.

The file here: huffman.cpp (C++)

Some screenshots:

The problem: Animal guessing game
What I did, the source: in C++

Thought of using classes but was short of time heh!

I AM IN DIRE NEED OF A THUMBDRIVE. Why I didn’t try to get one all this while is beyond me. It’s not that expensive. I guess I was just too lazy to go these IT places alone. But oh my god do I need one. Ok before I get carried away by the absence of a tiny cheap gadget, it’s already March people! Just when I thought I still have 12 months to go…

Long break really knocks me back at work – I spent half day figuring what everyone else was doing the past week so I could write the report. And then some. Urghhh!!! Now I’m in between controllers and models, trying out to figure out what did I do last week. I hate leaving on a spur like that because I didn’t do enough mental notes to connect to when I get back to it.

It is now 3.30pm, 2 more hours I’ll rush out of here. I don’t know, sometimes it happens to people I guess. There are days when the stars are just not aligned, the moon is not so bright, the wind seems stale, etc, you get the idea. Oh well at least I get a week break which is never a bad thing. Only now I have started to realized how busy I’m going to be the rest of this year.

Already given the assignment on computer networks. I never liked networks — eeek! Binary tree and such is such a welcoming balance to this semester. I don’t really care how computers are interconnected, just get me a working one, well connected one and let me build the apps ok. You worry about setting up the infrastructure, I worry about how to store the data and manipulate them. Isn’t that sound nicer. There’s network programming paper coming up – wonder whether or not will I like it. We’ll see.

I’m rambling.

Better go now before I start saying more nonsensical stuff.

On last note: I have this sudden, an immense liking to bachata music. Just perfect to listen to at night, especially if you have that special someone. Warning: if you don’t like latin music, don’t bother. You have been warned – you are prohibited to cringe upon listening.

Song 1 – Ven Tu

Listen to it while you can, I’m going to take it down in a few days and after that you’re on your own tralalalaa ~~