Introduction to Data Ingestion
Ingesting from APIs
Reading Multiple File Formats
Pydantic for Data Validation
Writing to Databases
Error Handling and Logging
Practice
Assignment
Gotchas & Pitfalls
Back to Track
8. Assignment
Content coming soon...
Suggested Project Scope
- Ingest data from a public API (or provided CSV files)
- Parse and normalize the data into a consistent format
- Validate records using Pydantic models, reporting invalid records
- Write clean records to Postgres
- Generate an error report (JSON or CSV) of failed validations
- Deliverables: working ingestion script, Postgres database with data, error report, documentation
- Requirements: handle at least 2 data sources or formats, implement pagination or chunking
Back to sidebar
*https://hackyourfuture.net/*
Found a mistake or have a suggestion? Let us know in the feedback form.