Week 3 - Ingesting and Validating Data

Introduction to Data Ingestion

Ingesting from APIs

Reading Multiple File Formats

Pydantic for Data Validation

Writing to Databases

Error Handling and Logging

Practice

Assignment

Gotchas & Pitfalls

• Teachers

Back to Track

2. Production Error Handling

Content coming soon...

Suggested Topics

• Error handling patterns for data ingestion (try/except for API, file, DB operations)
• Distinguishing transient vs permanent errors (retry vs fail)
• Retry strategies: exponential backoff, max attempts, circuit breakers
• Collecting validation errors: don't fail fast, accumulate all failures
• Writing error reports: failed records to error tables or CSV files
• Transactional rollback: undoing partial writes when a batch fails
• Graceful degradation: continuing processing despite some errors
• Using Week 1 logging patterns to log these errors
• Error tracking in production: recording which records failed and why
• Example: API call fails → log error → retry with backoff → eventually fail gracefully

Back to sidebar

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.