Assignment: Build Two Dashboards
This page is optional. Nothing here is required for Week 11's learning goals or the assignment. It traces how the dashboard became the last mile of the data stack, from 1980s spreadsheets to today's Python data apps. Read it in one sitting, or come back when you wonder why Metabase and Streamlit feel so different.
Week 10's History of Analytics Engineering covers how data got modelled, and Week 12's History of Data Orchestration covers how it got moved on a schedule. This page covers the arc in between: how modelled data got shown to people.
Background for Introduction to Dashboarding
The first self-serve analytics tool was the spreadsheet. VisiCalc shipped in 1979, Lotus 1-2-3 in 1983, and Microsoft Excel in 1985. For the first time, a non-programmer could load numbers into a grid, write a formula, and see a result update instantly. Every dashboard filter you build today is a descendant of that grid: change an input, watch the outputs recompute.
In 1993, the database researcher Edgar Codd coined the term OLAP (online analytical processing) to describe querying data along multiple dimensions at once: revenue by region by month by product. The mental model was the data cube, and the verbs were "slice", "dice", "drill down", and "roll up". Those verbs are still printed on the buttons of every BI tool.
For you as a data engineer, this era explains the shape of a mart table. When you built fct_daily_borough_stats in Week 10 with one row per borough per date, you were pre-computing a slice of a cube so a dashboard could read it instantly instead of aggregating millions of trips on every page load.
<aside> 🤓 Curious Geek: The spreadsheet that sold the computer
VisiCalc was the first "killer app": a piece of software so useful that people bought the hardware just to run it. Businesses bought the Apple II, a $2,000 machine, specifically to run a $100 spreadsheet. The pattern repeats constantly in data tooling: the analysis people want pulls the infrastructure into the building.
</aside>
Background for Introduction to Dashboarding
Through the 1990s and 2000s, business intelligence meant a small number of expensive enterprise platforms: Cognos (BI products from the early 1990s, bought by IBM in 2008), BusinessObjects (founded 1990, bought by SAP in 2007), MicroStrategy (founded 1989), and Hyperion. A single licence ran into six figures in euros per year, the software was installed and owned by a central IT team, and getting a new report often meant filing a ticket and waiting weeks.
This is the world "self-serve" was a reaction against. It also explains two habits you practised this week. The read-only metabase_user you connected with descends from the strict access governance these platforms enforced: analysts could read the warehouse but never write to it. And the careful metric definitions you wrote come from the same era's pain, where two departments would each build a "revenue" report off different assumptions and then argue about whose number was right in the quarterly meeting.
For you as a data engineer, the lesson is that the governance and definition discipline outlived the expensive tools. You apply it now with a free Metabase and a Markdown file, but the reasons are unchanged.
Background for Dashboards in Metabase
In 2003, a Stanford research project became Tableau, built on a query language called VizQL that turned drag-and-drop gestures into database queries and charts. Alongside Sweden's QlikView, Tableau made data visualisation a craft in its own right: the choice of chart type, colour, and layout became part of the analyst's job, not an afterthought. Tableau went public in 2013 and was acquired by Salesforce in 2019.
Two more tools defined the 2010s. Microsoft shipped Power BI in 2015, bundling Tableau-style visuals with the Office ecosystem and aggressive pricing, which is why it now dominates Dutch analyst job postings. Looker, founded in 2012, took a different path: it introduced LookML, a version-controlled modelling layer that defined metrics once so every chart computed them the same way. Google acquired Looker in 2019.
For you as a data engineer, the Question-then-Dashboard workflow you used in Metabase is the direct descendant of this wave. The idea that a saved query carries its own chart type, and that dashboards are arrangements of those saved queries, was normalised by Tableau and Looker before Metabase adopted it.
Background for Metabase Setup
Around 2015 a wave of open-source BI tools broke the price barrier. Metabase (2015) aimed at non-technical self-serve. Apache Superset was created at Airbnb in 2015 and donated to the Apache Foundation. Redash (2013, later acquired by Databricks) focused on SQL-first teams. Suddenly a startup could stand up a dashboard layer for the cost of a small server instead of a six-figure licence, and could read the source code to see exactly how a number was computed.
Open-source BI also normalised embedding: dropping a live chart into another product or an internal wiki through a public or signed URL, which is the share mechanism you used on your Metabase dashboard. The trade-off you met in the gotchas, that a public link exposes whatever the connected database user can read, is the same trade-off every embedded-BI team manages in production.
For you as a data engineer, this is why the class can run a real BI tool at all: the HYF-managed Metabase exists because someone could deploy it for free and you can inspect its behaviour rather than trust a black box.
<aside> 🤓 Curious Geek: One engineer, two tools you used this term
Apache Superset was created by Maxime Beauchemin at Airbnb. A few years earlier, at the same company, he had created Apache Airflow, the orchestrator you meet in Week 12. The modern open-source data stack was shaped by a surprisingly small group of people scratching their own itches.
</aside>
Background for Streamlit Fundamentals and Building a Metrics Dashboard
BI tools answer "what is the business doing?" for stakeholders. But engineers needed something else: a quick, code-first way to build a custom view with arbitrary Python logic, like calling an API or computing a freshness check. The answer came from a different lineage. R users got Shiny in 2012. Plotly released Dash in 2017. Then Streamlit, open-sourced in 2019 and acquired by Snowflake in 2022, made the pattern trivial with its script-rerun model: write a plain Python script top to bottom, and the whole thing re-executes on every interaction.
That rerun model is exactly why @st.cache_data mattered in your metrics app. Without it, every widget click would re-hit Postgres. The model trades a little inefficiency for a huge gain in simplicity, no callbacks, no component tree, which is what makes Streamlit the right tool for an engineer who wants a metrics panel in twenty minutes, not a polished stakeholder product.
For you as a data engineer, the split you practised this week, point-and-click Metabase for stakeholders and code-first Streamlit when you need custom logic, is the split this lineage created. The two tools come from different problems, which is why they feel different to use.
Background for Presenting Metrics
The oldest problem in this whole history, two people computing "revenue" differently, never fully went away. Looker's LookML was an early answer: define a metric once, in version control, and let every chart reference it. The idea matured into the semantic layer, and in 2023 dbt Labs acquired MetricFlow and shipped the dbt Semantic Layer, putting metric definitions next to the SQL models that produce them.
This is the production-grade home for the five-field metric definitions you wrote by hand this week. The Markdown file is the training-wheels version; in a real project, the "Calculation" field lives in your dbt schema.yml so that changing the SQL and changing the definition happen in the same pull request. That co-location is the only durable fix for the metric drift described in Presenting Metrics.
For you as a data engineer, this is the through-line: the dashboard is the last mile, but the trustworthy number behind it is defined upstream, close to the data. The presentation layer changed completely from 1990 to today; the discipline of defining the metric did not.
| Era | Technology | What it solved | What it left behind |
|---|---|---|---|
| 1979-1993 | Spreadsheets, OLAP cubes | Self-serve "what if" analysis for non-programmers | Single-user files, no shared source of truth |
| 1990-2008 | Cognos, BusinessObjects, MicroStrategy | Governed enterprise reporting | Six-figure licences, IT-owned, slow to change |
| 2003-2019 | Tableau, Power BI, Looker | Visualisation as a craft; the semantic layer | Mostly proprietary and licence-gated |
| 2013-2015 | Metabase, Superset, Redash | Free, inspectable, embeddable BI | Smaller built-in ecosystems than the incumbents |
| 2012-2019 | Shiny, Dash, Streamlit | Code-first custom dashboards for engineers | Not self-serve; needs Python to change anything |
| 2012-2023 | LookML, dbt Semantic Layer | Define a metric once, next to the data | Still maturing; tool-specific syntax |
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.