Hi there,
i'm new to this forum .. Maybe my way of expression is not very good, but I hope to be understandable.
I've a sql server 2005 database with 90 columns and more or less 185 thousands records. I've to run microsoft associations rules on my laptop (sony vaio sz3, core 2 duo, 2gb ram).
The problems is that the amount of ram seems not to be enough [it starts to swap when it's reading 240th case)
Because of this, i decided to sample my data by extracting 10thousands records randomnly ... it lasts 25minutes (more or less) now, but it's still to much...
Does a better way exists? What's the problem: column or row numbers?
hope you guys can help me.
There is a known issue with certain association rules models running out of memory but it's not very common (A fix is available through Microsoft Support).
The pointers in this thread might also be useful: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1613732&SiteID=1
Can you post your mining structure/model definition?
|||The problem with association rules is usually the parameters you set on the algorithm.
Try this - first create the structure/model and deploy without processing. Then to a process Structure Only to verify that there isn't a problem is just accessing the data.
If that works fine, then set the model parameter MINIMUM_SUPPORT very high - say 0.80. This means that only items that occur in 80% of the cases will be counted. Since you have dense data this could be an issue. Then you can gradually lower your minimum support to see how it impacts performance and your results. Additionally, since you do have dense data (e.g. all attributes on a single row) it is possible that you have some data that simply isn't interesting because it is true for everyone or almost everyone. You can set MAXIMUM_SUPPORT to a number lower than 1, for example setting it to 0.9 will eliminate all items and itemsets that occur in 90% or more of cases.
HTH
-Jamie
No comments:
Post a Comment