Pivot - An experiment from Microsoft Live Labs

Pivot is a way to interact with massive amounts of data, available at this link. It requires Silverlight.
Pivot makes it easier to interact with massive amounts of data in ways that are powerful, informative, and fun. We tried to step back and design an interaction model that accommodates the complexity and scale of information rather than the traditional structure of the Web.
When we use the Web today we treat the most fundamental scenarios as separate activities. Search takes us from many things to one, browsing moves us from one thing to another, and recommendations expose affinities that enable us to explore related topics. Can we do better by combining these scenarios into a more unified experience?
At the heart of Pivot are "Collections." They combine large groups of similar items on the Internet, so we can begin viewing the relationships between individual pieces of information in a new way. By visualizing hidden patterns, Pivot enables users to discover new insights while interacting with thousands of things at once.

mashzone.com - Help out in your business

Create attractive business mashups in a matter of minutes with MashZone and gain insights into your corporate processes and how they interrelate. Obtain the details you need to make informed decisions. Create your own mashups that are linked directly to current company data. Obtain a proper overview by combining data from different sources. Share your mashups with colleagues. Decide who can view your mashups and who can edit them. Benefit from exchanging knowledge with other users.

Interact with your data like never before. Bridge the gap between static tables, evaluations, and rigid reports and the need for graphical, interactive, easy-to-understand presentations. Create your own custom dashboard using a simple drag-and-drop interface. Drill down to detailed information via the interactive visualization options to display relationships between data.

Read more at this site (link)

Learn software concepts - Introduction to Data Binding

Data binding is the process that establishes a connection between the application UI and business logic. If the binding has the correct settings and the data provides the proper notifications, then, when the data changes its value, the elements that are bound to the data reflect changes automatically. Data binding can also mean that if an outer representation of the data in an element changes, then the underlying data can be automatically updated to reflect the change. A typical use of data binding is to place server or local configuration data into forms or other UI controls.

Basic Data Binding Concepts :
Data binding is based on a component architecture that consists of four major pieces : the data source object (DSO), data consumers, the binding agent, and the table repetition agent. Data source objects provide the data to a page, data-consuming HTML elements display the data, and the agents ensure that both the provider and the consumer are synchronized.

Direction of the Data Flow :
The data flow of a binding can go from the binding target to the binding source and/or from the binding source to the binding target.
- One Way binding causes changes to the source property to automatically update the target property, but changes to the target property are not propagated back to the source property. This type of binding is appropriate if the control being bound is implicitly read-only.
- Two Way binding causes changes to either the source property or the target property to automatically update the other. This type of binding is appropriate for editable forms or other fully-interactive UI scenarios. Most properties default to One Way binding, but some dependency properties default to Two Way binding.
- OneWayToSource is the reverse of OneWay binding; it updates the source property when the target property changes. One example scenario is if you only need to re-evaluate the source value from the UI.

Data Source Objects
To bind data to the elements of an HTML page in Windows Internet Explorer, a DSO must be present on that page. DSOs implement an open specification that leaves it up to the DSO developer to determine the following:
- How the data is transmitted to the page. A DSO can use any transport protocol it chooses. This might be a standard Internet protocol, such as HTTP or simple file I/O. A DSO also determines whether the transmission occurs synchronously or asynchronously. Asynchronous transmission is preferred, because it provides the most immediate interactivity to the user.
- How the data set is specified. A DSO might require an Open Database Connectivity (ODBC) connection string and an Structured Query Language (SQL) statement, or it might accept a simple URL.
- How the data is manipulated through scripts. Since the DSO maintains the data on the client, it also manages how the data is sorted and filtered.
- Whether updates are allowed.

Data Consumers
Data consumers are elements on the HTML page that are capable of rendering the data supplied by a DSO. Elements include many of those intrinsic to HTML, as well as custom objects implemented as Java applets or Microsoft ActiveX Controls.
A DSO typically exposes this functionality through an object model that is accessible to scripts.

Binding Agents
The binding and repetition agents are implemented by MSHTML.dll, the HTML viewer for Internet Explorer, and they work completely behind the scenes. When a page is first loaded, the binding agent finds the DSOs and the data consumers among those elements on the page. Once the binding agent recognizes all DSOs and data consumers, it maintains the synchronization of the data that flows between them.

Introduction to Database Concurrency

DATABASE CONCURRENCY: - Database concurrency is the particular situation when a single database is being accessed by multiple programs. Databases, by design in most cases are shared resources, but in this case, they are shared across multiple applications.
Database concurrency controls ensure that transactions occur in an ordered fashion. The main job of these controls is to protect transactions issued by different users/applications from the effects of each other. They must preserve the four characteristics of database transactions: atomicity, isolation, consistency and durability. Concurrency control is one of the main issues in the study of real time database systems. In addition to satisfying consistency requirements as in traditional database systems, a real time transaction processing system must also satisfy timing constraints.

Conflicts between transactions can be detected in two ways.
Pessimistic method detects conflicts before making access to the data object. When a transaction requests access to some data item, the concurrency control manager will examine this request and will determine whether to grant the request or not.
Optimistic schemes are designed to get rid of the locking overhead. They are optimistic in the sense that they take into account the explicit assumption that conflicts among transactions are rare events. The task of concurrency control is deferred until the end of transaction when some checking for potential conflicts has to take place and will be resolved accordingly, taking into consideration the amount of progress that has been done and the nature of conflict with transactions.
When concurrency control detects a conflict among some concurrent transactions accessing the same object, a conflict resolution mechanism needs to be put on. Concurrency control manager decides which transaction (victim) to penalize (the lock holder or the requester) and chooses an appropriate action and suitable timing. Two possible actions are most used: Blocking (wait) and abort (restart). In pessimistic concurrency control either blocking or abort can be used to resolve the conflict. However, in optimistic concurrency control only aborting is appropriate since conflict has been detected after the transaction has accessed the data object and performed some computation.

OPTIMISTIC CONCURRENCY CONTROL : The basic idea of an optimistic concurrency control mechanism is that the execution of a transaction consists of three phases: read, validation and write phases. For all optimistic concurrency control (OCC) schemes a conflict is detected after the data object has been accessed. In the OCC, conflict detection and resolution are both done at the certification time when a transaction completes its execution; it requests the concurrency control manager to validate all its accessed data objects. If it has not yet been marked for abort, it enters the commit phase where it writes all its updates to the database.

Learn software concepts: Introduction to Database Encryption

Encryption can provide strong security for data, but is that enough ? Data in a database can be accessed by many systems, but developing a database encryption strategy must take many factors into consideration. Where should the encryption be performed, for example — in the database, or in the application where the data originates? Who should have access to the encryption keys? How much data must be encrypted to provide security? What’s an acceptable trade-off between data security and application performance?
Data encryption is a process of converting stored or transmitted data to a coded form in order to prevent it from being read by unauthorized person. It is an application of a specific algorithm to alter the appearance of data, making it incomprehensible to those who are not authorized to see the information.
There are 2 types of encryption algorithm: -
- Secret key or Symmetric key algorithm: -In this encryption algorithm, a single secret or private key is shared between the sender and receiver. The sender encrypts this using this key and receiver decrypts it using the same key. It is highly assumed that no one else knows the key.
- Public key or Asymmetric key algorithm: - In this algorithm, every sender and receiver has a pair of keys. One is made public to the network and called public key and the other is kept private to that node called private key. The pair is such made that if the data is encrypted with one of the keys in the pair, it can only be decrypted with other key in the pair. When a sender has to send, it encrypts the data with receiver’s public key & the receiver decrypts it with its private key.

Advice on how to overcome some of the challenges in database encryption:
- Regulatory drivers : Advanced security through database encryption is required across many different sectors, and increasingly to comply with regulatory mandates.
One approach that can help companies address the encryption challenges associated with regulation is the defense-in-depth principle which advocates many layers to strong security – ranging from physical security and access controls to rights assignment and network security, including firewalls and, crucially, encryption of both data at rest and in transit.
- Overcoming key management issues
It is important that database encryption be accompanied by key management; however, statistics show that this is also the main barrier to database encryption. It is well-recognized that key use should be restricted and that key backup is extremely important. An additional best practice rule of encryption is that the encrypted key should never be stored alongside the data it was used to encrypt. Placing encryption keys within the HSM enforces this policy.
- Separation of duties and dual control
Many organizations pay close attention to separation of duties and dual control, which is required to pass audits to show that there are internal controls protecting against rogue administrators or unauthorized employees and is often required by the various regulatory requirements discussed above.

Database Integrity

Database Integrity is the preservation of data correctly & implies the process of keeping the dbase away from accidental deletion or alteration.
There are following types of integrity constraints:-
• Entity integrity constraints
• Referential integrity constraints
• Domain integrity constraints

DATABASE SECURITY: - Database security is a measurement of confidence that the integrity of a system and its data will be preserved.
Database security is assigned to address the following issues:-
• Privacy of data elements
• Preserving policies of organization
• System related security level
• Maintaining integrity of the database

Data integrity can be compromised in a number of ways:
- Human errors when data is entered.
- Errors that occur when data is transmitted from one computer to another.
- Software bugs or viruses.
- Hardware malfunctions, such as disk crashes.
- Natural disasters, such as fires and floods.

There are many ways to minimize these threats to data integrity. These include:
- Backing up data regularly.
- Controlling access to data via security mechanisms.
- Designing user interfaces that prevent the input of invalid data.
- Using error detection and correction software when transmitting data.

* Declarative Ease
Define integrity constraints using SQL statements. For these reasons, declarative integrity constraints are preferable to application code and database triggers. The declarative approach is also better than using stored procedures, because the stored procedure solution to data integrity controls data access, but integrity constraints do not eliminate the flexibility of ad hoc data access.
* Centralized Rules
Integrity constraints are defined for tables (not an application) and are stored in the data dictionary. Any data entered by any application must adhere to the same integrity constraints associated with the table.
* Maximum Application Development Productivity
If a business rule enforced by an integrity constraint changes, then the administrator need only change that integrity constraint and all applications automatically adhere to the modified constraint.
* Superior Performance
The semantics of integrity constraint declarations are clearly defined, and performance optimizations are implemented for each specific declarative rule.
* Flexibility for Data Loads and Identification of Integrity Violations
You can disable integrity constraints temporarily so that large amounts of data can be loaded without the overhead of constraint checking.
* The Performance Cost of Integrity Constraints
The advantages of enforcing data integrity rules come with some loss in performance.

Overview of Database Security

Database security is the set of systems, processes, and procedures that protect a database from unintended activity. Unintended activity can be categorized as authenticated misuse, malicious attacks or inadvertent mistakes made by authorized individuals or processes. Database security is also a specialty within the broader discipline of computer security. The database is the entity where all the data is stored, so protecting it from unauthorized access and change is extremely critical.
Traditionally databases have been protected from external connections by firewalls or routers on the network perimeter with the database environment existing on the internal network opposed to being located within a demilitarized zone.
Database security can begin with the process of creation and publishing of appropriate security standards for the database environment. The standards may include specific controls for the various relevant database platforms; a set of best practices that cross over the platforms; and linkages of the standards to higher level polices and governmental regulations.
One of the easiest steps to take is regarding passwords. Default or weak passwords are still often used by enterprises to protect an online asset as important as a database, but it's a problem that's easy to fix. The remedy is enforcing a strong password policy; that is, passwords must be changed regularly and be at least 10 digits long and contain both alphanumeric characters and symbols.
SQL Injection attacks pose tremendous risks to web applications that depend upon a database back-end to generate dynamic content. In this type of attack, hackers manipulate a web application in an attempt to inject their own SQL commands into those issued by the database. To prevent this type of attack, it is essential to ensure that all user-supplied data is validated before letting it anywhere near your scripts, data access routines and SQL queries, and preferably use parametrized queries. Another reason to validate and clean data received from users is to prevent cross-site scripting (XSS) attacks, which can be used to compromise a database connected to a Web server.
A database security program should include the regular review of permissions granted to individually owned accounts and accounts used by automated processes. The accounts used by automated processes should have appropriate controls around password storage such as sufficient encryption and access controls to reduce the risk of compromise.
The software used for the database, for the middle layers and for all other layers should be updated regularly with patches, updates and fixes. Falling behind in this task is pretty painful if you end up exposing holes in the software to attackers (and attackers know that a number of companies do not upgrade their systems on an immediate basis).

Introduction to Distributed Database Systems

A distributed database appears to a user as a single database but is, in fact, a set of databases stored on multiple computers. The data on several computers can be simultaneously accessed and modified using a network. Each database server in the distributed database is controlled by its local DBMS, and each cooperates to maintain the consistency of the global database.

Clients, Servers, and Nodes :
A database server is the software managing a database, and a client is an application that requests information from a server. Each computer in a system is a node. A node in a distributed database system can be a client, a server, or both.
A client can connect directly or indirectly to a database server.


Site Autonomy :
Site autonomy means that each server participating in a distributed database is administered independently (for security and backup operations) from the other databases, as though each database was a non-distributed database. Although all the databases can work together, they are distinct, separate repositories of data and are administered individually. Some of the benefits of site autonomy are as follows:
- Nodes of the system can mirror the logical organization of companies or cooperating organizations that need to maintain an "arms length" relationship.
- Local data is controlled by the local database administrator. Therefore, each database administrator's domain of responsibility is smaller and more manageable.
- Independent failures are less likely to disrupt other nodes of the distributed database. The global database is partially available as long as one database and the network are available; no single database failure need halt all global operations or be a performance bottleneck.
- Failure recovery is usually performed on an individual node basis.
- A data dictionary exists for each local database.
- Nodes can upgrade software independently.

Homogenous Distributed Database Systems :
A homogenous distributed database system is a network of two or more Oracle databases that reside on one or more machines. An application can simultaneously access or modify the data in several databases in a single distributed environment. For example, a single query on local database MFG can retrieve joined data from the PRODUCTS table on the local database and the DEPT table on the remote HQ database.


Heterogeneous Distributed Database Systems :
In a heterogeneous distributed database system, at least one of the databases is a non-Oracle system. To the application, the heterogeneous distributed database system appears as a single, local, Oracle database; the local Oracle server hides the distribution and heterogeneity of the data.
The Oracle server accesses the non-Oracle system using Oracle8i Heterogeneous Services and a system-specific transparent gateway. For example, if you include a DB2 database in an Oracle distributed system, you need to obtain a DB2-specific transparent gateway so that the Oracle databases in the system can communicate with it.

Introduction to Data Mining

Data mining is the process of discovering meaningful new correlations, patterns and trends by sifting through large amounts of data stored in repositories, using pattern recognition technologies as well as statistical and mathematical techniques.
Data mining is the analysis of (often large) observational data sets to find
unsuspected relationships and to summarize the data in novel ways that are
both understandable and useful to the data owner.
Data mining is an interdisciplinary field bringing together techniques from
machine learning, pattern recognition, statistics, databases, and visualization to
address the issue of information extraction from large data bases.

WHY MINE DATA ?
1. Commercial View :
- Lots of data is being collected and warehoused.
* Web data, e-commerce.
* Purchases at department/grocery stores.
* Bank/Credit Card transactions.
- Computers have become cheaper and more powerful
* Competitive Pressure is strong.
* Provide better, customized services for an edge.
2. Scientific View :
- Data collected and stored at enormous speeds(GB/hour).
* Remote sensors on a satellite.
* Telescopes scanning the skies.
* Micro arrays generating gene expression data.
* Scientific simulations generating terabytes of data.
- Traditional techniques infeasible for raw data.
- Data mining may help scientists :
* in classifying and segmenting data.
* in Hypothesis Formation.

SCOPE OF DATA MINING :
Data mining derives its name from the similarities between searching for valuable business information in a large database — for example, finding linked products in gigabytes of store scanner data — and mining a mountain for a vein of valuable ore. Both processes require either sifting through an immense amount of material, or intelligently probing it to find exactly where the value resides. Given databases of sufficient size and quality, data mining technology can generate new business opportunities by providing these capabilities:
- Automated prediction of trends and behaviors.
- Automated discovery of previously unknown patterns.

Automated discovery of previously unknown patterns.
* More columns : Analysts must often limit the number of variables they examine when doing hands-on analysis due to time constraints. Yet variables that are discarded because they seem unimportant may carry information about unknown patterns. High performance data mining allows users to explore the full depth of a database, without preselect a subset of variables.
* More rows : Larger samples yield lower estimation errors and variance, and allow users to make inferences about small but important segments of a population.

Introduction to Databases

Databases play an important role in almost all areas where they are used including business, engineering, medicine, law, education, and library science, to name a few.
A database is a collection of related data, where data means recorded facts. A typical database represents some aspect of the real world and is used for specific purposes by one or more groups of users. Databases are not specific to computers. Examples of non-computerized databases abound: phone book, dictionaries, almanacs, etc. A database has the following implicit properties :
1. A database represents some aspect of the real world.
2. A database is a logically coherent collection of data with some inherent meaning.
3. A databse is designed, built, and populated with data for a specific purpose.
4. A databse can be of any size and of varying complexity.
5. A database may be generated and maintained manually or it may be computerized.
A database management system (DBMS) is a collection of programs that enables users to create and maintain a database. The DBMS is a general-purpose software system that facilitates the process of defining, construction, and manipulating databases for different applications.
Defining a database involves specifying the data types, structures, and constraints for the data to be stored in the database.
Constructing a database is the process of storing the data itself on some storage medium that is controlled by the DBMS.
Manipulating a database includes such functions as querying the database to retrieve specific data, updating the database and generating the reports from the data.



CHARACTERSTICS THAT DISTINGUISH DATABASE APPROACH FROM TRADITIONAL FILE-PROCESSING APPLICATIONS :
- Existence of a catalog : It contains information such as structure of each file, the type and storage format of each data item and various constraints on the data. The information stored in catalog is called meta-data.
- Program data independence : In traditional file processing, the structure of a file is embedded in the access programs, so any changes to the structure of a file may require changing all programs that access this file. By contrast, the structure of data files is stored in DBMS catalog separately from access programs. This property is called program data independence.
- Program operation independence: Users can define operations on data as part of database applications. An operation is specified in two parts - interface of operation : includes operation name and data types of its arguments, implementation of operation : specified separately and can be changed without affecting the interface. This is called
program operation independence.
- Data abstraction : The characteristic that allows program data independence and program operation independence is called data abstraction.
- Support of multiple user views.
- Sharing of data among multiple transactions.

Main Categories of Database users are :
- Administrators.
- Designers.
- End users.
- System analysts and application programmers.
- DBMS system designers and implementers.
- Tool Developers.
- Operators and maintenance personnel.

Advantages of using Databases :
- Potential for enforcing standards.
- Reduced application development time.
- Flexibility.
- Availability of up-to-date information to all users.
- Economies of sale.

Problems with depending on cloud computing

In recent years, we are being sold on the total promise of cloud computing, or to be on a much simpler level, storing our data on sites on the internet and depend on internet applications for a lot of their work. Some examples of these are using Google Mail, Hotmail, Google Docs, Online Maps, Online photo sharing and storage. Corporations also depend on applications running off the internet such as Salesforce, Google Apps, etc. In fact, the entire concept of Software as a Service (SAAS) depends on companies basing their primary business applications on 3rd party hosted apps. We are now at that stage when companies no longer have a backup for these services; consider your own case - when you save something on Google Docs, do you also have a local copy of that data ? Do you have a backup way of running your business when the internet app goes down for whatever reason ?
Most companies now depend on these hosted services / data storage being always available. After all, if you are a photo storage company and depend on customer photos being stored on Amazon's S3 service, the service better be always available. If the service even goes down for a couple of hours, that is a time when your customers can no longer access their photos, and would not be a pleasant experience. Now consider the recent case of Google services being unavailable for a few hours due to a traffic jam at one of its data centers. This means that services such as Google Analytics, Gmail, Google Docs, Maps, and so on were unavailable (link to article):

Google has apologized for yesterday's service outage that left 14 percent of its user base without Google's wide variety of online services for a few hours. Google said in a blog post the outage came down to a simple traffic jam at an Asian data center. Well, a quick look at this graph from the Web security company Arbor Networks shows a canyon-sized hole in North American Internet traffic during the G-outage. With a wide variety of practical services like Gmail, Google Docs, Maps, Calendar, and even Google search gone, online activities came to a standstill for many people during the Google blackout.
Just how smart is it to depend on a company to store all your data online? Some smaller storage companies have even gone under without giving users a chance to collect their precious bits and bytes. Canadian photographer Ryan Pyle told Spring how he lost more than 7000 edited and retouched images after the storage company Digital Railroad abruptly shut its doors last year.

Leads to 2 problems - with many companies operating on wafer thin margins and in a recession, there is a greater chance of many companies disappearing. If these companies were in the service of either data storage or app hosting, then customers will be hit when these companies go down. In some cases, when the disappearance is sudden, then customers may be hit with data loss.
In the second case, it may be possible that a company does not go down, but operations are hit for some time due to some technical issues, planned downtime, or even hacker attacks. Customers dependent on them will need to suspend activities during such a period.

Roadkil's Disk Wipe

Securely erases the contents of a disk replacing it with random data or leaving the drive completely blank. Numerous passes can be performed to ensure data is totally unrecoverable. Program works on hard and floppy disks aswell as USB/Flash drives.

Windows XP, Windows Vista, Windows NT, Windows 9x/Me, Windows 2000

Download from this location (link)

DiskScrub - Hard Drive Data Elimination Software

It is becoming standard practice in corporations, government agencies, law firms and accounting firms to reassign computers and to donate older computers to charity. Millions of personal computers have been put to use since 1981 when the IBM Personal Computer came into existence. Many of the older personal computers have been reassigned or donated to charity and many more will fall into this category in the future. However, data security is often ignored when computers change hands.
Personal computers were never designed with security in mind. Potentially anything that transpired on a used computer still exists. Multiply that by the number of computers your organization will reassign or surplus this year, and you get the point. Computers should be reassigned and donated to charity but the contents of the hard disk drives should not be ignored.
With computer technology changing almost daily, corporations and government agencies have to stay current while still making the best uses of aging computer resources. Advancements in hard disk drive storage capacities, operating systems and software applications cause corporations to buy or lease new computers every year. But what is done with the old computers? What is done about the sensitive data still existing, essentially "stored" on these computers when they are sold, transferred or donated? That is a serious problem, and NTI's Disk Scrub software was specifically designed to deal with these risks, for corporations, government agencies, hospitals, financial institutions, law firms and accounting firms.

Read more at this location (link)

Erase Hard Drives, Certified Data Erasure

Compliance regulations (HIPAA, Sarbanes Oxley SOX, etc.) mandate the secure erasure of all files and data that are subject to federal regulation. Failure to prevent unauthorized disclosure of privileged information can result in serious civil and criminal liability.
CyberScrub provides a wide array of hard drive erasure solutions. These products use erase methods that meet and exceed requirements set by the US Department of Defense (US DoD 5220.22) to ensure total data destruction.
cyberCide is designed for full disk sanitization, and may be deployed locally or over the network. Detailed log files, providing hard drive serial numbers, dates/times of erasure, erase method used, technician, bad sectors and other data is generated to be parsed into your compliance, audit and inventory reports. CyberCide may be run from command-lines and modified per your specifications as applicable.

Read more and download from the site (link)

Mozy - Backup system

Chances are you have a lot of important stuff on your computer like financial documents, email, digital photos, music and more. Unfortunately, computers are vulnerable to hard drive crashes, virus attacks, theft and natural disasters, which can erase everything in an instant. Current statistics show that one in every ten hard drives fail each year. The cost of recovering a failed hard drive can exceed $7,500, and success is never guaranteed.
Mozy is a simple and safe way to back up all the important stuff on your computer. A copy of your data is stored in a secure, remote location for safekeeping, so that in the event of disaster your data is still retrievable. Mozy makes online backup possible for everyone with an affordable, secure solution that's easy to use.

Read more at this location.