Posts Tagged ‘postgrad’

I’m on semester break!

// January 21st, 2010 // 2 Comments » // life

I suddenly found this huge amount of time. I mean really HUGE. I have week nights and weekends free. Well maybe not to you, but it is that huge if you’re studying part time. I was so used to worry about assignments and studying every moment every day, now for the next 3 months I am free! So I’ve been this free only for a week, I got my room cleaned up – got it into much neater state (heh) by bringing in 3 new proper furnitures – a bed, a study table and a wardrobe. Now everything is in its place and happy.

I’m not going to take pictures of my room (sorry Kak Watiey heheh) – it’s such a sacred place for me. But they are very similar like these, except mine all in dark wood colour:



Ok that’s done and now last night I started looking at Django + PostgreSQL + Python in general. Was kinda slow, I have this habit of looking too far ahead (maybe it’s thinking too much) so I was worried that I would ruin my laptop in the future by blindly install everything inside. Typical woman? Being paranoid of nothing? Yea jumping in too early is not my thing, I’m scared if anything goes wrong. Laptop dies, application breaks, stuff happens yes I know but I don’t want it to be because I didn’t do my best to prevent it from happening.

Anyway, yea I’m embracing this temporary new found freedom – will put it to good use. Was so tempting to watch TV (I LOVE WATCHING TV – enough said) at nights but like my mom says “TV will be there forever waiting for you, go do something else better” so no worries there. Besides, I like programming. It’s like a drug. I remember Harish Pillay said during FOSS.my 2009, “How do you think an author wins an award for his/her book? He/She probably reads alot of other books to write it. So, how do you be good in programming? You read alot of codes.”

I love that man.

Now I know how to mine data

// January 5th, 2010 // 1 Comment » // misc, open source

There’s just too many techniques on how you can do data mining. In grad school we only got to know 1 technique – by using artificial neural network. There’s a tool for this – Weka. Download and install it or sudo apt-get it (yes imagine my surprise when I found out about this).

What’s data mining?

Well, basically it’s a process to get a knowledge (eg. if customers buy diapers, most likely they will buy beers too) from your data (imagine spreadsheets filled with numbers and strings). You may have thousands of records but how do you make sense of it? You mine those data, and get a pattern. From there you’ll get a knowledge. So from this you can put beers next to diapers in supermarkets to increase sales. That’s the general idea.

There’s a 90% chance that the data you got is not cleaned aka there’s missing values, some data are not consistent (eg. in sex can only have F or M, there’s a value Q?), and some data are just plain rubbish. The data need to go through preprocessing stage. For missing values, you have to filled them up, either using mean or median values, whichever is best for your data. The same goes to inconsistent data. This is where working with experts in the domain you’re working in is very important. You don’t want to remove what you think is rubbish but it actually means something.

Then we have data discretization process where you reduce that huge amount of data but they still carry the same value. Afterwards we normalize the data. After this is done, then the data is ready for the modelling exercise.

Classification

One of the benefit you get from data mining is classification. It’s where you can predict if you have this data (eg. male, married, doesn’t have children, have regularly purchase beer for the past year) whether beer purchase is likely. So it will classify to something like this purchase_beer equals to 0 (no) or 1 (yes). In order for the model to predict 0 or 1, the model itself has to be trained with alot of data. Train it until it reaches the accuracy we want, above 90% is good. A trained model with very high accuracy is going to be an asset as you can feed it data and it will spit out what we want.

The most difficult part in doing this for me is the data preprocessing part. You have to have quality cleaned data to produce quality results. You have to carefully select which kind of data is relevant to your goal (eg. would you want to include one’s job as one of the attributes considered for the diapers-beers example?) which is why having domain experts is important. They also have to determine how each data should affect the result (eg. job probably affects 10% but marital status affects 80% towards diapers-beers purchase).

I just love AI

There are just so many preprocessing methods & data mining hybrid techniques already been researched by academicians. I’m just so overwhelmed by the amount of technical papers on this. They probably not so much IT savvy like us, the implementors as they call it, but they definitely have the brain on that part of the world. We just have to scour through this massive database and get it to run on our apps. Well, should you need it that is. Coz processing thousands of data can take hours or days, some months, depending on your machine spec.

Open data

While doing data mining assignment last 2 weeks, I became frustrated with the unavailability of data in Malaysia. Sad. Maybe it’s still difficult for us to see it now, I’m already imagining all the stuff we can do with those data like in JPA, MOHR, MOH, MOHE – fuhhhh I’m all shuddery now. Even in OSCC, those training feedback forms and MyGOSSCON feedback forms – hmm whatever happened to those?

There are concerns of exposing private data I guess. Well, if you ask me make the data anonymous, as I couldn’t care less who got promoted last year. I only want the ’spec’ of that person who got promoted – age, sex, location, department, salary, is he respected by peers, does he drive, etc, that kind of stuff, you get the idea.

Hmm.. this will take another 10 years to realise, I think.

I’ve had pretty good lecturers so far at UKM

// December 18th, 2009 // No Comments » // life

Now that I’ve just found out the thing that we’re supposed to submit tomorrow is now postponed to next week, I’m taking a moment to write about my 4 lecturers I’ve had so far and my whole experience being a in grad school here.

Semester I
The conditions of the class were pretty common and just above tolerable. No power points in sight except at the very front and the very back. When it was raining I prayed to god that I won’t die because of the cold temperature. Being in an IT faculty, taking an IT course, but no wifi? The ones they had was the prepaid ones. I couldn’t be bothered to search for any open shops in campus that’s open on Saturdays. Being one of the research university, free wifi should be available to all staff and students! Unless I need that rare publications, I can pretty much do anything on the internet.

Dr. Md Jan taught Data Structures (we got to know trees). He’s a considerate person. It’s a very straightforward paper, like Maths, that I loved it! He’s always open for discussions, jovial and committed. I like him because he made use of internet in his lectures. He’s current. He let us choose our own programming language for his assignments. Being an open source person, that’s freedom of choice and I liked it.

Prof. Patel taught Computer Network. I just couldn’t bring myself to like this paper. It felt like out of this world. He came across as someone who teaches just to pay the rent. Nothing’s wrong there of course, it just felt ordinary. I couldn’t understand his assignments, I did it just to pass the test. Network is something that I don’t wish to dwell in so much.

Semester 2
Hey the classroom looks so much better. New floor, new chairs so I the chilling problem gone. They set up power points! And there’s wifi! Applaudable effort for me even though it’s not stable all the time but the thought is there. Keep it up!

Dr. Salwani teaches Software Engineering. Her expertise is in AI but she used to be a developer so it’s actually captivating to listen to her lectures (at least for me coz I can relate). She used real life scenarios and real life problems & solutions. I think she’s well respected in her field to other academicians in Malaysia. She’s got a thing with open source that I’m not so comfortable with (from past experience maybe?). She’s tough and kinda strict but lacks a bit in managing assignments deadlines. We have 2 assignments still not sure of the deadlines. I think my classmates wish she would just take back the those 2 since other one we’ve been doing is quite huge!

Dr. Azura teaches Advanced Artificial Intelligent. She’s one of the top AI researchers along with Dr. Salwani, although her interest is in data mining. The work they’ve done, I mean, seriously, big corporations should hire/consult these 2 – they can give you patterns and predict things! I am astounded that I had not looked into AI before (I studied maths doing my first degree and I though AI is very physics-like haha thanks to tv shows), that I feel those CS graduates should all work in AI industry! Don’t just program a program, program an AI program. Anyway, she’s a great person, abit chatty in the classroom although I don’t mind it since the class is 3-5pm, and a very knowlegable person.

Conclusion?
Although the lecturers have been great, I’ve had some problems with my group assignments. I like if the assignments is individual coz I don’t have to rely on others to do my work. I have no problems with working in a group, it’s just frustrating some of them like to work last minute. So I usually did my part early and trust that they would take that and continue. When it’s time to submit, it’s disheartening looking at the end result so messy and unorganized. Oh well.

Being the only programmer in the group really sucks sometimes. The group did their part, I believe as much as they could but I get the right to grumble. Non-programming assignment, I contribute to the group. When it comes to programming, only me doing it? It’s because they have already forgetten how to program. This is real, so real that it sucks being a programmer in this case.

Studying really takes a toll in your life. Your priority changes abit. Not much if you’re single, alot if you’re married. I understand my group have husbands and children to care for but it’s not good if you abandon your team on not one, but many presentation days. My advice to married women who wants to pursue study:

[1] Let everyone in your family know your intention. If you’re studying, let them know you’re studying. Your priority has changed abit. Last time you go straight to the kitchen after work 5 times a week, now you only do it 2 times a week. There’s just some routine stuff you can’t do anymore.
[2] Your husband should understand what being a grad student means. It doesn’t mean you still can be with him everytime going to everywhere, not until you finish studying. It doesn’t mean he can continue old habit of not doing anything around the house after work (just an example). You wife has other stuff to do now, give a take a little now can we? If you can’t work together, I assure you, you’re going to be staying up late every night just to catch up with the lessons.
[3] Please understand the sacrifice you have to make if you’re studying. You’re going to be tired, busy all the time and you can’t afford to be at all family occasions/kenduri kawen kawan anymore. If you’re studying, you want to be good at it and because it means something to you. If not why do it at all?

Anyway, I actually enjoy the whole period so far although I don’t have time for anything else. You want it, you have to sacrifice your time. I now know some stuff that I probably wouldn’t have known if I’m learning on my own. Granted there are times I felt like quitting – but that’s just normal I guess.

Education is not cheap these days. I should squeeze everything out of it so I get the money worth. All in all I think UKM is an ok place for me. As long as they have good lecturers, that’s the most important thing and so far I think they have pretty good ones at FTSM.

Fuzzy reasoning & Artificial Neural Network

// November 17th, 2009 // 2 Comments » // misc

Interesting assignment for my AI paper. The first one asks for me to calculate someone’s learning difficulty level, given his IQ level and his recent test score (education domain). Another one asks to predict the level of H1N1 risk of a patient, given his severity of these symptoms: fever, breathing difficulty, fatique and coughing.

Fuzzy reasoning is useful when you’re dealing with ‘high IQ’, ‘average IQ’, ‘low IQ’, etc. How would you translate ‘high’ or ‘average’ into crisp numbers like more than 160 it’s ‘high IQ’. If it’s more that 130cm it’s ‘average IQ’. Of course the numbers can’t be used for everyone (I may have a different opinion on how much is ‘high IQ’). And you can’t use simple algorithm if and else like we normally would do. Fuzzy sets on the other hand allows you to have 160 falls under ‘high IQ’ and ‘average IQ’ at say, 0.8 and 0.6 (on a scale 0 to 1) each. Once you have defined all that, you have to have your fuzzy rules to work on. Given someone’s IQ level and his test score, you can calculate his learning difficulty level and then maybe you can have this result to propose a new set of tests that match his level.

Artificial neural network emulates how the brain works, well sort of. It has the capability to learn and predict stuff. Like the H1N1 case, the model will learn to reach a very good predict whether the patient has high risk of being infected with H1N1 or not. It is much dependant on number of cases, the more cases you feed the model, the better the result will be.

Personally I think it’s all mathematics. Everywhere you see calculations. Once you understand how it works, it’s pretty easy (the data I worked on is not that much hahaha). In real world, the data set is huge, so data mining is actually intriguing, I think. Worth knowing.

In the risk of my classmates meniru, I’m going to upload what I have done so far, which I doubt that they’ll find it here until much later. If you manage to get it here anyway, hey I really don’t mind you looking around. Just be kind enough to give feedbacks there’s mistakes in the doc :P .

Okay. HUNGRY. AND SLEEPY.

Grad school (Part 1): AI is blowing me away!

// August 29th, 2009 // 1 Comment » // misc

Well, so far at least. So today marks the beginning of my second semester here in UKM. I can’t wait for this nightmare to end. 3 more sems to go… hinnnnssssss hinnsss hinss..

Anyway, the AI course – I like the lecturer already, she encourages us to use open source in doing assignments. And she’ll tailored the assignments to fit the background of her students. One thing that I like is that she’ll fail anyone who doesn’t submit assignments even though it carries 40%.
(more…)