Introduction to Data Ingestion
Ingesting from APIs
Reading Multiple File Formats
Pydantic for Data Validation
Writing to Databases
Error Handling and Logging
Practice
Assignment
Gotchas & Pitfalls
Back to Track
6. Writing to Databases
Content coming soon...
Suggested Topics
- Local Postgres setup: installation, basic commands
- Connecting to Postgres from Python:
psycopg2, SQLAlchemy
- Creating tables: schema definition, primary keys, constraints
- Inserting records: single vs bulk inserts, performance implications
- Raw tables pattern: storing ingested data as-is before processing
- Upserts (INSERT...ON CONFLICT): idempotent inserts
- Transaction management: commit, rollback, handling failures
- Connection pooling: managing database connections efficiently
- Error handling: duplicate keys, constraint violations, connection loss
- Querying what you wrote: validation that data made it to the database
Back to sidebar
*https://hackyourfuture.net/*
Found a mistake or have a suggestion? Let us know in the feedback form.