Recently, I was dealing with a task where I had to import raw information into a staging PostgreSQL Database as part of a task in the Airflow DAG pipeline. I would like to share the approach used and exposes the valuable Pandas.json_normalize function and how it helps to prepare the data before importing. Reading the…
Category: Data Engineering
Getting the last modified directory with Python
Working with os paths is a prevalent task, especially when working on a console application in Python. Recently, I had to add a change to an existing feature where it was mandatory to return the last modified directory on a specific target. In Python, we have the opportunity to count with different libraries; in this…
First steps for configuring SQL Server with Docker
Sometimes, our working habits could experiment some interesting changes, maybe forced by real needs or even simply for the valid desire to explore new ways to do some tasks. I remember in the past, I used to install SQL Server Developer edition on my laptop f. Nonetheless, I have had to add other Databases such…
An introduction to pathlib module of Python
First at all, I would like to clarify something that maybe sounds evident, but sometimes we forget, the path is not a string, it is a fact that we must keep in mind to avoid headaches in our journey in Python. Since Python 3.4 in advance, we count on a powerful and versatile module called…