Data Mining: What is Data Mining?
Overview
Generally, data mining (sometimes called data or knowledge discovery) is the process of analyzing data from different perspectives and summarizing it into useful information - information that can be used to increase revenue, cuts costs, or both. Data mining software is one of a number of analytical tools for analyzing data. It allows users to analyze data from many different dimensions or angles, categorize it, and summarize the relationships identified. Technically, data mining is the process of finding correlations or patterns among dozens of fields in large relational databases.
Continuous Innovation
Although data mining is a relatively new term, the technology is not. Companies have used powerful computers to sift through volumes of supermarket scanner data and analyze market research reports for years. However, continuous innovations in computer processing power, disk storage, and statistical software are dramatically increasing the accuracy of analysis while driving down the cost.
Example
For example, one Midwest grocery chain used the data mining capacity of Oracle software to analyze local buying patterns. They discovered that when men bought diapers on Thursdays and Saturdays, they also tended to buy beer. Further analysis showed that these shoppers typically did their weekly grocery shopping on Saturdays. On Thursdays, however, they only bought a few items. The retailer concluded that they purchased the beer to have it available for the upcoming weekend. The grocery chain could use this newly discovered information in various ways to increase revenue. For example, they could move the beer display closer to the diaper display. And, they could make sure beer and diapers were sold at full price on Thursdays.
Data, Information, and Knowledge
Data
Data are any facts, numbers, or text that can be processed by a computer. Today, organizations are accumulating vast and growing amounts of data in different formats and different databases. This includes:
* operational or transactional data such as, sales, cost, inventory, payroll, and accounting
* nonoperational data, such as industry sales, forecast data, and macro economic data
* meta data - data about the data itself, such as logical database design or data dictionary definitions
Information
The patterns, associations, or relationships among all this data can provide information. For example, analysis of retail point of sale transaction data can yield information on which products are selling and when.
Knowledge
Information can be converted into knowledge about historical patterns and future trends. For example, summary information on retail supermarket sales can be analyzed in light of promotional efforts to provide knowledge of consumer buying behavior. Thus, a manufacturer or retailer could determine which items are most susceptible to promotional efforts.
Data Warehouses
Dramatic advances in data capture, processing power, data transmission, and storage capabilities are enabling organizations to integrate their various databases into data warehouses. Data warehousing is defined as a process of centralized data management and retrieval. Data warehousing, like data mining, is a relatively new term although the concept itself has been around for years. Data warehousing represents an ideal vision of maintaining a central repository of all organizational data. Centralization of data is needed to maximize user access and analysis. Dramatic technological advances are making this vision a reality for many companies. And, equally dramatic advances in data analysis software are allowing users to access this data freely. The data analysis software is what
supports data mining.
What can data mining do?
Data mining is primarily used today by companies with a strong consumer focus - retail, financial, communication, and marketing organizations. It enables these companies to determine relationships among "internal" factors such as price, product positioning, or staff skills, and "external" factors such as economic indicators, competition, and customer demographics. And, it enables them to determine the impact on sales, customer satisfaction, and corporate profits. Finally, it enables them to "drill down" into summary information to view detail transactional data.
With data mining, a retailer could use point-of-sale records of customer purchases to send targeted promotions based on an individual's purchase history. By mining demographic data from comment or warranty cards, the retailer could develop products and promotions to appeal to specific customer segments.
For example, Blockbuster Entertainment mines its video rental history database to recommend rentals to individual customers. American Express can suggest products to its cardholders based on analysis of their monthly expenditures.
WalMart is pioneering massive data mining to transform its supplier relationships. WalMart captures point-of-sale transactions from over 2,900 stores in 6 countries and continuously transmits this data to its massive 7.5 terabyte Teradata data warehouse. WalMart allows more than 3,500 suppliers, to access data on their products and perform data analyses. These suppliers use this data to identify customer buying patterns at the store display level. They use this information to manage local store inventory and identify new merchandising opportunities. In 1995, WalMart computers processed over 1 million complex data queries.
The National Basketball Association (NBA) is exploring a data mining application that can be used in conjunction with image recordings of basketball games. The Advanced Scout software analyzes the movements of players to help coaches orchestrate plays and strategies. For example, an analysis of the play-by-play sheet of the game played between the New York Knicks and the Cleveland Cavaliers on January 6, 1995 reveals that when Mark Price played the Guard position, John Williams attempted four jump shots and made each one! Advanced Scout not only finds this pattern, but explains that it is interesting because it differs considerably from the average shooting percentage of 49.30% for the Cavaliers during that game.
By using the NBA universal clock, a coach can automatically bring up the video clips showing each of the jump shots attempted by Williams with Price on the floor, without needing to comb through hours of video footage. Those clips show a very successful pick-and-roll play in which Price draws the Knick's defense and then finds Williams for an open jump shot.
How does data mining work?
While large-scale information technology has been evolving separate transaction and analytical systems, data mining provides the link between the two. Data mining software analyzes relationships and patterns in stored transaction data based on open-ended user queries. Several types of analytical software are available: statistical, machine learning, and neural networks. Generally, any of four types of relationships are sought:
* Classes: Stored data is used to locate data in predetermined groups. For example, a restaurant chain could mine customer purchase data to determine when customers visit and what they typically order. This information could be used to increase traffic by having daily specials.
* Clusters: Data items are grouped according to logical relationships or consumer preferences. For example, data can be mined to identify market segments or consumer affinities.
* Associations: Data can be mined to identify associations. The beer-diaper example is an example of associative mining.
* Sequential patterns: Data is mined to anticipate behavior patterns and trends. For example, an outdoor equipment retailer could predict the likelihood of a backpack being purchased based on a consumer's purchase of sleeping bags and hiking shoes.
Data mining consists of five major elements:
* Extract, transform, and load transaction data onto the data warehouse system.
* Store and manage the data in a multidimensional database system.
* Provide data access to business analysts and information technology professionals.
* Analyze the data by application software.
* Present the data in a useful format, such as a graph or table.
Different levels of analysis are available:
* Artificial neural networks: Non-linear predictive models that learn through training and resemble biological neural networks in structure.
* Genetic algorithms: Optimization techniques that use processes such as genetic combination, mutation, and natural selection in a design based on the concepts of natural evolution.
* Decision trees: Tree-shaped structures that represent sets of decisions. These decisions generate rules for the classification of a dataset. Specific decision tree methods include Classification and Regression Trees (CART) and Chi Square Automatic Interaction Detection (CHAID) . CART and CHAID are decision tree techniques used for classification of a dataset. They provide a set of rules that you can apply to a new (unclassified) dataset to predict which records will have a given outcome. CART segments a dataset by creating 2-way splits while CHAID segments using chi square tests to create multi-way splits. CART typically requires less data preparation than CHAID.
* Nearest neighbor method: A technique that classifies each record in a dataset based on a combination of the classes of the k record(s) most similar to it in a historical dataset (where k 1). Sometimes called the k-nearest neighbor technique.
* Rule induction: The extraction of useful if-then rules from data based on statistical significance.
* Data visualization: The visual interpretation of complex relationships in multidimensional data. Graphics tools are used to illustrate data relationships.
What technological infrastructure is required?
Today, data mining applications are available on all size systems for mainframe, client/server, and PC platforms. System prices range from several thousand dollars for the smallest applications up to $1 million a terabyte for the largest. Enterprise-wide applications generally range in size from 10 gigabytes to over 11 terabytes. NCR has the capacity to deliver applications exceeding 100 terabytes. There are two critical technological drivers:
* Size of the database: the more data being processed and maintained, the more powerful the system required.
* Query complexity: the more complex the queries and the greater the number of queries being processed, the more powerful the system required.
Relational database storage and management technology is adequate for many data mining applications less than 50 gigabytes. However, this infrastructure needs to be significantly enhanced to support larger applications. Some vendors have added extensive indexing capabilities to improve query performance. Others use new hardware architectures such as Massively Parallel Processors (MPP) to achieve order-of-magnitude improvements in query time. For example, MPP systems from NCR link hundreds of high-speed Pentium processors to achieve performance levels exceeding those of the largest supercomputers.
Don’t believe one optimistic word from any public figure about the economy or humanity in general. They are all part of the problem. Its like a game of Monopoly. In America, the richest 1% now hold 1/2 OF ALL UNITED STATES WEALTH. Unlike ‘lesser’ estimates, this includes all stocks, bonds, cash, and material assets held by America’s richest 1%. Even that filthy pig Oprah acknowledged that it was at about 50% in 2006. Naturally, she put her own ‘humanitarian’ spin on it. Calling attention to her own ‘good will’. WHAT A DISGUSTING HYPOCRITE SLOB. THE RICHEST 1% HAVE LITERALLY MADE WORLD PROSPERITY ABSOLUTELY IMPOSSIBLE. Don’t fall for any of their ‘humanitarian’ CRAP. ITS A SHAM. THESE PEOPLE ARE CAUSING THE SAME PROBLEMS THEY PRETEND TO CARE ABOUT. Ask any professor of economics. Money does not grow on trees. The government can’t just print up more on a whim. At any given time, there is a relative limit to the wealth within ANY economy of ANY size. So when too much wealth accumulates at the top, the middle class slip further into debt and the lower class further into poverty. A similar rule applies worldwide. The world’s richest 1% now own over 40% of ALL WORLD WEALTH. This is EVEN AFTER you account for all of this ‘good will’ ‘humanitarian’ BS from celebrities and executives. ITS A SHAM. As they get richer and richer, less wealth is left circulating beneath them. This is the single greatest underlying cause for the current US recession. The middle class can no longer afford to sustain their share of the economy..... Their wealth has been gradually transfered to the richest 1%. One way or another, we suffer because of their incredible greed. We are talking about TRILLIONS of dollars. Transfered FROM US TO THEM. Over a period of about 27 years. Thats Reaganomics for you. The wealth does not ‘trickle down’ as we were told it would. It just accumulates at the top. Shrinking the middle class and expanding the lower class. Causing a domino effect of socio-economic problems. But the rich will never stop. They will never settle for a reasonable share of ANYTHING. They will do whatever it takes to get even richer. Leaving even less of the pie for the other 99% of us to share. At the same time, they throw back a few tax deductible crumbs and call themselves ‘humanitarians’. Cashing in on the PR and getting even richer the following year. IT CAN’T WORK THIS WAY. Their bogus efforts to make the world a better place can not possibly succeed. Any 'humanitarian' progress made in one area will be lost in another. EVERY SINGLE TIME. IT ABSOLUTELY CAN NOT WORK THIS WAY. This is going to end just like a game of Monopoly. The current US recession will drag on for years and lead into the worst US depression of all time. The richest 1% will live like royalty while the rest of us fight over jobs, food, and gasoline. Crime, poverty, and suicide will skyrocket. So don’t fall for all of this PR CRAP from Hollywood, Pro Sports, and Wall Street PIGS. ITS A SHAM. Remember: They are filthy rich EVEN AFTER their tax deductible contributions. Greedy pigs. Now, we are headed for the worst economic and cultural crisis of all time. SEND A “THANK YOU” NOTE TO YOUR FAVORITE MILLIONAIRE. ITS THEIR FAULT. I’m not discounting other factors like China, sub-prime, or gas prices. But all of those factors combined still pale in comparison to that HUGE transfer of wealth to the rich. Anyway, those other factors are all related and further aggrivated because of GREED. If it weren’t for the OBSCENE distribution of wealth within our country, there never would have been such a market for sub-prime to begin with. Which by the way, was another trick whipped up by greedy bankers and executives. IT MAKES THEM RICHER. The credit industry has been ENDORSED by people like Oprah, Ellen, Dr Phil, and many other celebrities. IT MAKES THEM RICHER. Now, there are commercial ties between nearly every industry and every public figure. IT MAKES THEM RICHER. So don’t fall for their ‘good will’ BS. ITS A LIE. If you fall for it, then you’re a fool.. If you see any real difference between the moral character of a celebrity, politician, attorney, or executive, then you’re a fool. WAKE UP PEOPLE. THEIR GOAL IS TO WIN THE GAME. The 1% club will always say or do whatever it takes to get as rich as possible. Without the slightest regard for anything or anyone but themselves. Reaganomics. Their idea. Loans from China.. Their idea. NAFTA. Their idea. Outsourcing. Their idea. Sub-prime. Their idea. The commercial lobbyist. Their idea. The multi-million dollar lawsuit.. Their idea. $200 cell phone bills. Their idea. $200 basketball shoes. Their idea. $30 late fees. Their idea. $30 NSF fees. Their idea. $20 DVDs. Their idea. Subliminal advertising. Their idea. Brainwash plots on TV. Their idea... Prozac, Zanex, Vioxx, and Celebrex. Their idea. The MASSIVE campaign to turn every American into a brainwashed, credit card, pharmaceutical, love-sick, couch potatoe, celebrity junkie. Their idea. All of the above shrink the middle class, concentrate the world’s wealth and resources, and wreak havok on society. All of which have been CREATED AND ENDORSED by celebrities, athletes, executives, entrepreneurs, attorneys, and politicians. IT MAKES THEM RICHER. So don’t fall for any of their ‘good will’ ‘humanitarian’ BS. ITS A SHAM. NOTHING BUT TAX DEDUCTIBLE PR CRAP.. In many cases, the 'charitable' contribution is almost entirely offset.. Not to mention the opportunity to plug their name, image, product, and 'good will' all at once. IT MAKES THEM RICHER. These filthy pigs even have the nerve to throw a fit and spin up a misleading defense with regard to 'tax revenue'. ITS A SHAM. THEY SCREWED UP THE EQUATION TO BEGIN WITH. ITS THEIR OWN DAMN FAULT. If the middle and lower classes had a greater share of the pie, they could easily cover a greater share of the federal tax revenue. They are held down in many ways because of greed. Wages remain stagnant for millions because the executives, celebrities, athletes, attorneys, and entrepreneurs, are paid millions. They over-sell, over-charge, under-pay, outsource, cut jobs, and benefits to increase their bottom line. As their profits rise, so do the stock values. Which are owned primarily by the richest 5%. As more United States wealth rises to the top, the middle and lower classes inevitably suffer. This reduces the potential tax reveue drawn from those brackets. At the same time, it wreaks havok on middle and lower class communities and increases the need for financial aid.. Not to mention the spike in crime because of it. There is a dominoe effect to consider. So when people forgive the rich for all of the above and then praise them for paying a greater share of the FEDERAL income taxes, its like nails on a chalk board. If these filthy pigs want to be over-paid, then they should be over-taxed as well. Remember: The richest 1% STILL own 1/2 of all United States wealth EVEN AFTER taxes, charity, and PR CRAP. A similar rule applies worldwide. There is nothing anyone can say to justify that. Anyway, there is usually a higher state and local burden on the middle class. They get little or nothing without a local tax increase. Otherwise, the red inks flows like a waterfall. Service cuts and lay-offs follow. Again, because of the OBSCENE distribution of bottom line wealth in this country. I can not accept any theory that our economy would suffer in any way with a more reasonable distribution of wealth. Afterall, it was more reasonable 30 years ago. Before Reaganomics came along. Before GREED became such an epidemic. Before we had an army of over-paid executives, celebrities, athletes, attorneys, investors, entrepreneurs, developers, and sold-out politicians to kiss their asses. As a nation, we were in much better shape. Lower crime rate, more widespread prosperity, stable job market, free and clear assets, lower deficit, ect. Our economy as a whole was much more stable and prosperous for the majority. WITHOUT LOANS FROM CHINA. Now, we have a more obscene distribution of bottom line wealth than ever before. We have a sold-out government, crumbling infrastructure, energy crisis, home forclosure epidemic, 13 figure national deficit, and 12 figure annual shortfall. ALL BECAUSE OF GREED. I really don't blame the 2nd -5th percentiles. No economy could ever function without some reasonable scale of personal wealth and income. But it can't be allowed to run wild like a mad dog. GREED KILLS. Bottom line: The richest 1% will soon tank the largest economy in the world. It will be like nothing we’ve ever seen before. and thats just the beginning. Greed will eventually tank every major economy in the world. Causing millions to suffer and die. Oprah, Angelina, Brad, Bono, Drew, and Bill are not part of the solution.. They are part of the problem. THERE IS NO SUCH THING AS A MULTI-MILLIONAIRE HUMANITARIAN. EXTREME WEALTH HAS MADE WORLD PROSPERITY ABSOLUTELY IMPOSSIBLE. WITHOUT WORLD PROSPERITY, THERE WILL NEVER BE WORLD PEACE OR ANYTHING EVEN CLOSE. GREED KILLS. IT WILL BE OUR DOWNFALL. Of course, the rich will throw a fit and call me a madman. Of course, their ignorant fans will do the same. You have to expect that. But I speak the truth. If you don’t believe me, then copy this entry and run it by any professor of economics or socio-economics. Then tell a friend. Call the local radio station. Re-post this entry or put it in your own words. Be one of the first to predict the worst economic and cultural crisis of all time and explain its cause. WE ARE IN BIG TROUBLE.