Summary - Metadata in data warehouses

Lecture



Это окончание невероятной информации про метаданные в хранилищах данных.

...

should describe the organization's lines of administrative and business activity, i.e., a description of the organization's business model;

  • a description of data quality. This fragment of the metadata model should describe the quality requirements for data being loaded into the DW;
  • a description of the DW's software and hardware environment. This fragment of the metadata model should describe the tables, columns, keys, and data processing processes in the DW;
  • a description of information security. This fragment of the metadata model describes the procedures for access control and user authentication in the system that uses access to the DW.

    • Note that this lecture has not paid any attention to the question of representing information security information. As a rule, software and hardware solutions in the area of information security are confidential in nature, and it is not advisable to give any general recommendations for describing them in a metadata model. This will be determined by the manager of the IT project creating the DW.

    Summary

    In this lecture we examined the concept of metadata as a set of specifications and data elements containing a description of the data of an information system and the processes for handling it. The basic functions were defined and a classification of metadata in a DW was given. A brief overview of the "Common Warehouse Metamodel" specification was given.

    Using the example of a specific data mart, it was shown in detail how to form metadata for the model, fact tables, facts, dimension tables, and data sources. The metadata description presented served as the basis for the logical design of the DW metadata model.

    Metadata is information about data that is required to manage a DW, and metadata management is an essential component of the warehouse architecture. Technical metadata includes all the information required to configure and use the DW. Subject-oriented metadata includes business terms and definitions of DW data. Structural metadata – is a description of DW objects and their characteristics. Data processing metadata is information collected during DW operation, such as the lineage of transferred and transformed data; the usage status of data (active, archived, or deleted); monitoring data such as usage statistics, error messages, and audit results.

    Metadata is often placed in a repository, which allows metadata to be shared among various tools and processes when designing, installing, applying, operating, and administering the DW.

    Продолжение:


    Часть 1 Metadata in data warehouses
    Часть 2 Metadata standards - Metadata in data warehouses
    Часть 3 Summary - Metadata in data warehouses

    Comments

    To leave a comment

    If you have any suggestion, idea, thanks or comment, feel free to write. We really value feedback and are glad to hear your opinion.
    To reply

    Lectures and tutorial on "Databases, knowledge and data warehousing. Big data, DBMS and SQL and noSQL"

    Terms: Databases, knowledge and data warehousing. Big data, DBMS and SQL and noSQL

    Database and knowledge. types of databases. relational and non relational
    Relational DB. types of normal data. Database normalization
    Distributed and parallel database systems
    Graph database NEO4J create, insert, delete, edit.Cypher
    Explaining how SQL JOIN works using the Venn diagrams as an example
    Relationships and associations between database tables: one-to-one, one-to-many, many-to-many
    Explaining SQL UNION using the Venn Diagrams as an Example
    Sets, relationships, fact table in relational databases
    Basic concepts of the relational data model
    4. Relational algebra - operations of union, intersection, difference, Cartesian product, selection, projection, join
    5. Elements of the SQL language
    6. Normal forms of relationships. Database Design, Types of Normal Forms
    7. Normal forms of higher orders
    8. Elements of the model "entity-relationship" ER-diagrams in databases
    9. Transactions and database integrity
    10. Transactions and concurrency
    11. Transactions and data recovery
    ORM object-relational mapping,
    MySQL InnoDB and MyISAM locks
    Blocking (DBMS) Pessimistic Optimistic Joint Exclusive Lowercase Granular
    questions and answers for the MYSQL interview in English
    1. The concept of a database
    2. Database structure
    3. Relational databases
    4. Conceptual database model
    5. Transformation of model into relational
    A few words about database design.
    Briefly about relational databases
    Infologic DB design. Entities, objects, properties, relations
    Data models Early approaches to data organization
    Database Design
    Storage Structures and Access Methods
    Types of subqueries (subqueries) in SQL
    Connections and set-theoretic relations on relations
    Requests using connections
    Theory of Data Synchronization in Computer Science
    Transaction isolation level
    index types Oracle, MySQL, PostgreSQL, MS SQL
    Organization of client-server data synchronization for a mobile application
    Multidimensional database technology
    Big Data Part 1: Principles of working with big data, the MapReduce paradigm
    Big Data Part 2: Hadoop
    Big data. Part 3: Techniques and strategies for developing MapReduce applications
    Big data. Part 3: Techniques and strategies for developing MapReduce applications
    Big Data from Part 4: Hbase
    Approaches to processing big data (Using only a detailed fact table, the method of chain interaction of documents, the relationship of tables through triggers)
    big data
    Big Data. Part 5.1: Hive - SQL engine over MapReduce
    Big Data Part 5.2: Advanced hive features
    The main functions of ETL-systems
    Statistical analysis in PostgreSQL using PL / R
    PostgreSQL - database overview, basics
    Multi-user database processing systems
    The concept and architecture of the distribution of databases. Homogeneous and heterogeneous raBD. Data Distribution Strategies in DBA
    Distributed DBMS (RDBMS). Twelve rules of K. Data. Advantages and disadvantages of RDBMS
    Types of data access interface (D) base
    The concept of data storage (DS). Differences between data warehouses and databases
    B. Inmon data warehousing classification
    Data Warehouse Technology Solutions . Key Data Warehouse Architecture Approaches
    Database users. Multi-user database problems. Database Administrator Functions
    Relevance of database protection. The causes of its destruction. Legal protection of databases
    Database recovery (using database backup, using transaction log, recovery via rollback, roll forward)
    Database Optimization
    Extract, transform, load (ETL)
    Persistent data structures
    Denormalization as a means of increasing productivity. Denormalization implementation
    In-memory database (IMDB)
    MVCC (multiversion concurrency control) — managing concurrent access through multiversioning
    Problems with slow ALTER TABLE in MySQL and possible solutions
    NoSQL (not only SQL)
    MySQL — a database overview: basics, purpose, and capabilities
    Ten myths about database performance
    Multi-Range Read (MRR) optimization in MySQL
    Codd's 12+1 rules for relational DBMS
    ACID (Atomicity, Consistency, Isolation, Durability) — requirements for a transactional system, and BASE
    Design and comparison of SQL and NoSQL databases
    Parametric and zone indexes
    Variants, capabilities, and specifics of sorting in MySQL
    WITH queries — Common Table Expressions (CTE), hierarchical and recursive queries in SQL
    The N + 1 selection problem in object-relational mapping (ORM)
    Kappa architecture: subtleties and meaning
    The need for and use of NULL in relational databases
    Common mistakes in writing SQL queries
    The CAP theorem and BASE architecture
    The object-relational gap and impedance mismatch
    SQL Antipatterns
    The entity-relationship modeling method
    The dimensional modeling method
    Temporal data modeling methods
    The Data Vault modeling method
    Relational data warehouses: concept, construction schemas, advantages and disadvantages
    Business Intelligence systems and data warehouses
    The physical model of a data warehouse
    Metadata in data warehouses
    The physical model of a data warehouse: accounting for the impact of transactions, table denormalization
    Designing data warehouse performance — indexing
    Aggregation and summarization in data warehouses
    Tuning the performance of data warehouse queries
    Designing data cubes
    Analytical data processing in data warehouses (analytic, statistical, ranking, window, and reporting functions)
    Building a data warehouse model based on the corporate data model
    Designing and developing the ETL process
    Window functions in SQL
    Architectures of remote databases
    Methods and types of database optimization: denormalization, indexes, buffer tables, replication, sharding, partitioning, lazy loading, MapReduce
    Algorithmic complexity (Big O) in SQL SELECT queries and with JOINs
    Full-text search and the full-text index
    Digital obsolescence
    Data types and their indexes in DBMS (MySQL, PostgreSQL, Microsoft SQL Server, Oracle Database)
    Ways of storing data inside relational databases
    Streaming data: collection, storage, processing, and examples
    Vector database — essence and examples