dlt-skill

The dlt-skill is a comprehensive assistant for data engineers and developers needing to implement robust data ingestion workflows. It automates the boilerplate of setting up dlt pipelines, ensuring that data is extracted, normalized, and loaded into your chosen data warehouse or lakehouse efficiently. Whether you are dealing with verified SaaS sources, standard REST APIs, or complex custom Python-based logic, this skill provides the decision frameworks, configuration templates, and operational scripts necessary to streamline the development lifecycle.

• Streamline dlt initialization with guided workflows for verified sources like Salesforce, GitHub, Stripe, and HubSpot.

• Implement declarative REST API pipelines for standard web services, including support for authentication, pagination, and JSON parsing.

• Develop custom Python extraction logic using @dlt.source and @dlt.resource decorators for specialized data sources or unique transformation requirements.

• Configure destination-specific settings for popular platforms such as Google BigQuery, Snowflake, and local DuckDB instances.

• Optimize data loading with built-in patterns for incremental loading, write dispositions (append, replace, merge), and schema evolution.

• Manage sensitive credentials via .dlt/secrets.toml and general configurations via .dlt/config.toml, with built-in guidance on security best practices like .gitignore.

• Facilitate observability using the built-in dlt dashboard for inspecting pipeline runs, schemas, and loaded records.

• Utilize the dlt-skill when facing tasks related to data ingestion, pipeline maintenance, dlt init commands, or debugging failed load jobs.

• Leverage the provided script library for automated dependency installation (supporting uv, pip, poetry, and pipenv) and dashboard launching.

• Inputs typically include API endpoints, database connection parameters, and desired data destinations; outputs include structured pipeline code, configuration files, and verified loading logs.

• Be aware that while verified sources simplify setup, custom Python sources provide unlimited flexibility but require manual maintenance of extraction logic and API error handling.

• Ensure that all secrets are properly managed within .dlt/secrets.toml and kept out of version control; follow the provided decision tree to determine the optimal pipeline approach before writing code.