QA Automation Engineer

At ITV, a Most Loved Workplace® certified employer in the Media & Entertainment space, Entertaining with purpose, connecting millions through creativity every day.

Data nerds, welcome to this data engineering bootcamp. We're over the next 27 hours, we're going to be mastering the two most in-demand skills. For this, we're going to start at the very beginning, covering the basics of core engineering concepts like the data engineering life cycle. And only then will we get into mapping how popular tools fit into this framework. From there, we'll shift into the most in-demand skill of SQL, applying engineering concepts to understand how this tool is used in the real world, while also learning popular skills like the cloud and bash. Finally, we'll get into learning the most popular programming language of Python, starting from the basics until you become a fluent coder, while also learning how to use tools like Git and GitHub. Now, this is not going to be 25 hours of me straight lecturing, we're actually going to be building something. You'll put together from scratch three portfolio projects to demonstrate your new found experience that you can use to showcase on your resume to an employer. So whether you're a complete beginner or switching careers and starting from zero with no degree or no experience, I built this for you. And all of this is completely free. I include all the resources and data necessary to complete the bootcamp in the description below. If you're new here, I'm Luke. And after gaining my engineering degree, I served the United States submarine force, spending a total of two years underwater. After that, I got my start working in data with a global fortune 500 company. I've made data courses not only on YouTube, but also for over 30,000 students on data camp. And my last job was with Mr. Beast building out data pipelines. Now, more recently, I've founded datanerd.tech. It's a real-time job market platform that's completely free. It not only tells you what are the top demanded skills for roles like data engineers and based on your home country, but it also goes a step further providing you with real-time job postings so you can apply. So we're going to be using the data for my app of over 4 million job postings in order to provide an unbiased opinion of what skills and what order you should be learning it to become a data engineer. But before we get into that, we need to actually understand what a data engineer actually does. So when I worked at that fortune 500 company, our data was spread all over the place. Our customer data was in Salesforce, which is a CRM tool or customer relations management tool. All of our product data was in a program called SAP, which is an ERP tool or enterprise resource planning. Oh, and I can never forget all those tribal knowledge Excel files that had pricing information. Needless to say, this is a mess, especially if you're trying to analyze your customer's buying pattern. You have to go to three different sources. Well, that's where data engineers come in. They take all that data spread across your source systems and build and maintain pipelines to move this data into a centralized system such as a data warehouse. And it's not just to move the data, but it's also to provide clean and reliable data, which then can be used to serve your different end consumers, such as data analysts, data scientists, and really anybody else that needs it. Now, there's three general roles that work in this type of ecosystem. The first are data engineers, which this is why you're on this video. They help build and maintain pipelines. Downstream of this are data analysts, which uses this data to inform business decisions. And then we also have data scientists, which use this data to build predictive models of what should happen. So in practice, what does this look like? Well, you'll be provided with a strategic business decision to solve, and you'll build a pipeline around this. Then data analysts will use tools like Excel and Power BI to generate reports and build dashboards to understand what is happening now. Similarly, data scientists will use tools like Python to build models in order to understand what is happening tomorrow. Now, if you want to dive deeper in understanding what a data engineer does, along with a lot of these concepts that we're going to cover in this video, I have a completely free data engineering crash course that's delivered directly to your inbox every morning for 10 days. This course covers all the essentials you need to know to land your first data engineering role. Check out the link in the description to sign up. Now, let's get into the curriculum of this bootcamp. And it's important to understand, I didn't just randomly choose Python and SQL. Instead, I chose them based on the data. If we go to my app, datanerd.tech, looking at the past year, there are 180,000 data engineering jobs analyzed for this, where each job is requesting on average seven skills and a pretty hefty salary of $140,000. Looking at how often a skill appears within a posting, we can see that by far SQL and then Python appearing in more than two out of three job postings. Now, there are other skills on the list, which we're going to get into covering in greater detail in the next video, but the key takeaway from this is that Python and SQL are non-negotiable compared to these other ones. So with that, you're probably wondering, what the heck is SQL? It also can be spoken SQL, and it stands for Structured Query Language. So how does this all work? Well, it starts with using this language to write a SQL query. That is then sent to a database, where your data is, and the query is executed inside of this. Finally, an output is provided that the task is complete, and this allows you to see what happened. Now, if SQL is how you talk to a database, Python is how you give your computer instructions to do, well, anything. Python is a multi-purpose programming language. It's used in engineering, while you're in this bootcamp. Also, analytics, machine learning, and automation. Now, comparing it to SQL, this is more specialized. It works in the database, creating, reading, updating, and deleting data. Also known as CRUD. Let's look into how Python works for an engineering task. You could write a Python script to run on your computer to execute an engineering workflow. For this, it could orchestrate multiple different SQL queries to perform an ELT, or extract, load, and transform process. These SQL queries themselves would execute inside of the database, because that's what SQL does. And from there, we'd have outputs for each step along the process. And what's great about Python is it can monitor each one of those outputs and ensure it's done properly. Here's something that tripped me up when I first started, and that's where does Python run? Well, you could run it locally, like on your computer, and it runs Python just fine. It's great for local testing and development. However, in production, you're typically running it on a remote computer, such as a server. And Python is a beast working on this. Now, getting into SQL, it's a little bit more limited. You can still run it on your local computer, but you need a database engine in order to execute your SQL query. And more popularly in production, in a server, you would have your database, and this is where you would be executing your SQL queries. And all of this ultimately hoped to demonstrate what skill between SQL and Python should you be learning first. Well, the core product that you're going to be building for your end consumers, it's going to revolve around databases. And databases require for you to know SQL in order for you to interact with that. Whereas in the case of something like Python, it's rarely, if at all, ever used alone in a data engineering project. Instead, it's most often used with SQL. So if it isn't clear, we're going to be covering SQL first and then Python. But there's actually a third secret weapon that I included in this. And that is that we're going to be using AI, specifically ChatGPT, to speed up our workflow and work more efficiently. Now, we're not only going to have a mini course on ChatGPT halfway through this, but I'll also be showing you my AI workflow as we go through both of these tools. All right, which gets into how the heck are we going to be learning both these tools? Well, I polled my subscribers asking them, what is the biggest impact in landing your first job? And projects by far were the number one method over degrees and certifications in landing their roles. So as you build the skills in this bootcamp, you're going to be also building concrete projects that you can showcase as experience. Let's get into the curriculum. We're going to start it all off with a crash course on the fundamentals, focusing heavily on the data engineering lifecycle. And then concepts like warehouses versus lake and lake houses, data modeling basics, and then ETL versus ELT. For SQL, we're going to start at the very beginning, assuming you have no coding experience. Getting into basic operations like queries, aggregations, and joins. We'll also beef up your engineering knowledge using different dev tools and jumping into data modeling. For the first project, we're going to build a pipeline performing an exploratory analysis of the data jobs postings from datanerd.tech. For the advanced section, we'll get into more complex queries like creating tables, along with concepts like subqueries, CTEs, window functions, and nested data. We'll also continue on our knowledge of engineering concepts with data modeling and dev tools like Git and GitHub. For the second project, we're going to build an end-to-end data warehouse. For this, we'll design a star schema along with different data marts that will be able to be served to a host of different data tools. After this, we'll jump into Python, once again, starting at the

Want to learn more about what it's like to work at ITV? View our full profile.