Featured Post

14 Top Data Pipeline Key Terms Explained

Image
 Here are some key terms commonly used in data pipelines 1. Data Sources Definition: Points where data originates (e.g., databases, APIs, files, IoT devices). Examples: Relational databases (PostgreSQL, MySQL), APIs, cloud storage (S3), streaming data (Kafka), and on-premise systems. 2. Data Ingestion Definition: The process of importing or collecting raw data from various sources into a system for processing or storage. Methods: Batch ingestion, real-time/streaming ingestion. 3. Data Transformation Definition: Modifying, cleaning, or enriching data to make it usable for analysis or storage. Examples: Data cleaning (removing duplicates, fixing missing values). Data enrichment (joining with other data sources). ETL (Extract, Transform, Load). ELT (Extract, Load, Transform). 4. Data Storage Definition: Locations where data is stored after ingestion and transformation. Types: Data Lakes: Store raw, unstructured, or semi-structured data (e.g., S3, Azure Data Lake). Data Warehous...

Top features in the design of data modelling (1 of 2)

Data modelling jobs and career
[Data modelling jobs career]
The analogy with architecture is particularly appropriate because architects are designers and data modeling is also a design activity. In design, we do not expect to find a single correct answer, although we will certainly be able to identify many that are patently incorrect. Two data modelers (or architects) given the same set of requirements may produce quite different solutions.

Data modeling is not just a simple process of "documenting requirements" though it is sometimes portrayed as such. Several factors contribute to the possibility of there being more than one workable model for most practical situations.

First, we have a choice of what symbols or codes we use to represent real-world facts in the database. A person's age could be represented by Birth Date, Age at Date of Policy Issue, or even by a code corresponding to a range ("H" could mean "born between 1961 and 1970").

Second, there is usually more than one way to organize (classify) data into tables and columns. In our insurance model, we might, for example, specify separate tables for personal customers and corporate customers, or for accident insurance policies and life insurance policies.

Third, the requirements from which we work in practice are usually incomplete, or at least loose enough to accommodate a variety of different solutions. Again, we have the analogy with architecture. Rather than the client specifying the exact size of each room, which would give the architect little choice, the client provides some broad objectives, and then evaluates the architect's suggestions in terms of how well those suggestions meet the objectives, and in terms of what else they offer.

Fourth, in designing an information system, we have some choice as to which part of the system will handle each business requirement. For example, we might decide to write the rule that policies of type E20 have a commission rate of 12% into the relevant programs rather than holding it as data in the database. Another option is to leave such a rule out of the computerized component of the system altogether and require the user to determine the appropriate value according to some externally specified (manual) procedure. Either of these decisions would affect the data model by altering what data needed to be included in the database.

Finally, and perhaps most importantly, new information systems seldom deliver value simply by automating the current way of doing things. For most organizations, the days of such "easy wins" have long passed. To exploit information technology fully, we generally need to change our business processes and the data required to support them. (There is no evidence to support the oft-stated view that data structures are intrinsically stable in the face of business change).The data modeler becomes a player in helping to design the new way of doing business, rather than merely reflecting the old.

Unfortunately, data modeling is not always recognized as being a design activity. The widespread use of the term "data analysis" as a synonym for data modeling has perhaps contributed to the confusion. The difference between analysis and design is sometimes characterized as one of description versus prescription.

We tend to think of analysts as being engaged in a search for truth rather than in the generation and evaluation of alternatives. No matter how inventive or creative they may need to be in carrying out the search, the ultimate aim is to arrive at the single correct answer. A classic example is the chemical analyst using a variety of techniques to determine the make-up of a compound.

Related:
Start training in Data modelling

Comments

Popular posts from this blog

How to Fix datetime Import Error in Python Quickly

SQL Query: 3 Methods for Calculating Cumulative SUM

Big Data: Top Cloud Computing Interview Questions (1 of 4)